Platform
Solution
Industry
Role
There is no result.

The Collector Debug Facility is a command-line interface (CLI) that you can use to remotely run debug commands on your Collector. This is helpful for troubleshooting issues with data collection. For example, if you wrote a script DataSource and are getting NaN values, or there is one instance out of ten  not reporting data, you can run the Collector Debug Facility to identify the issue.

Note: The history of Collector debug commands is preserved in the Audit Log. For more information, see Audit Logs Report.

You can run the Collector Debug Facility from the Collector you want to debug, or you can navigate to the DataSource instance in Resources. When you run the Collector Debug Facility, it opens in a new tab and you can run the debug commands applicable to your environment. A list of the debug commands along with their descriptions displays in the Collector Debug Facility interface to assist with troubleshooting prior to entering the commands you need.

collector debug console

Debug Commands for the Collector Debug Facility

The following table lists the debug commands you can run in the Collector Debug Facility for troubleshooting:

CommandDescriptionExample
!accountDisplays the account information used by sbwinproxy.!account
!adlistDisplays a list of the Collector’s Active Discovery tasks. A taskID is returned for each task.!adlist type=get

!adlist method=ad_snmp
!adetail <taskId>Displays detailed information about a specific Active Discovery task, where taskId can be found with !adlist.

Note: the “taskId” reference in the command specification will be labeled as “id” in the output of the !adlist command.

!adetail 142
!checkcredentialEnables, disables, and checks credential usage on the specified host to determine source of unexpected login action.!checkcrendential proto=snmp user=public

!checkcrendential proto=snmp user=public usage=AP
!DecryptFileSHAGets the SHA256 values of all libraries, DLL and JAR files which are exported when a collector is installed.
  • !DecryptFileSHA<filename>—Gets the decrypted SHA of the specified file.
  • !DecryptFileSHA—If a filename is not specified, the SHA of all files is displayed in the JSON format.
 		!DecryptFileSHA logicmonitor-util.jar
!hostpropertyAdds, updates, or deletes system properties for a host.hostproperty action=del host=localhost property=virtualization

!hostproperty action=add host=localhost property=ips value=127.0.0.1,192.168.1.1
!httpSends an HTTP request and displays the response. URL is required.

!http [username=xxx [password=yyy]] [followRedirect=true|false] [method=GET| POST| PUT] [version=1| 1.1] [timeout=<seconds>] url [-h "<header json>"] [-b "<body json>"]

The following are the optional arguments to display the response:

  • username=xxx [password=yyy]
    If your web endpoint requires Basic Authentication, you must include a user if you have a password. However, if you have a user, the password is not required.

  • followRedirect=true|false
    Determines whether or not to follow redirects

  • method=GET| POST| PUT
    Determines HTTP methodversion=1| 1.1
    Determines HTTP version

  • timeout=<seconds>
    Allows you to configure a custom timeout

  • -h “<header json>”
    Allows you to add custom headers

  • -b “<body json>”
    Sends a custom body (usually used with POST
!http http://www.google.com/index.html 


!http method=GET http://www.google.com/index.html

!http method=POST http://www.google.com/index.html -h "{"Content-type": "application/json"}" -b "{"key" : "value"}"

!http method=PUT http://www.google.com/index.html -h "{"Content-type": "application/json"}" -b "{"key" : "value"}"

!jdbcExecutes a SQL query against the specified host.!jdbc ‘url=jdbc:mysql://productrds.chqwqvn285rf.us-west-2.rds.amazonaws.com:3306 username=LogicMonitor password=MyPassword’ select Name, ID from productDB.Employees
!logsurfDisplays log file entries that are of the specified debug level. If included, logs will only be displayed for the specified seq and taskId if they are in the specified file, and only n number of logs will be displayed. taskId and seq can be found using !tlist, where taskId is the id of a data collection task and seq is the number of times the collector remembers having done the task.!logsurf level=trace ../logs/wrapper.log taskid=833 seq=75
!packetcapture2Captures the network traces of the device on which you are running this command.
If you do not pass an argument in the !packetcapture2 command, the system displays the following components:
  • Interface—The network interface on your system through which network traffic can be captured. Some common examples are:
    • Linux—eth0 (Ethernet), wlan0 (Wi-Fi), and lo (loopback)
    • Windows—interfaces are often named using a GUID, such as \Device\NPF_{GUID}
      !packetcapture2 uses either npcap (for Windows) or libpcap (for Linux) library. The device should have either npcap or libcap library installed on the machine. For example, for Linux machine, eth0 is one of the interfaces provided under libpcap library. Thus, you can specify (name=eth0,description=null).

    • File—The file name where all the network traces are saved. For example, test.pcap.
    • Timeout—All the network traces are captured for the specified time. The timeout is specified in minutes. If the specified timeout is extended, such as 50 minutes, then the request gets timed out, but the packet capture continues to run in the background and is saved in the file.
    • Filter—The filter captures all the network calls to the specified host and port. Some of the basic filters are:
    • "tcp"—Captures all the TCP packets
    • "udp"—Captures all the UDP packets
    • "icmp"—Captures all the ICMP packets
    • "ip"—Captures all the IP packets
    • "arp"—Captures all the ARP packets
    You can combine the filters to create new filters. For example, 
    • "tcp and port <port number>"—Captures the TCP packets of a specific port.
    • "udp and port <port number>"—Captures the UDP packets of a specific port.

    Downloading the .pcap file
    When the system captures network traces as per the specified filter, you can go to the file directory and view the result. For example,  /usr/local/logicmonitor/agent/logs/test.pcap.
    You can also download the .pcap file that has all the network traces saved in it. To do so, follow these steps:
    1. Enter the command !uploadlog followed by the file name. For example, !uploadlog test.pcap.
    2. Run the command. A ZIP file is downloaded.
!packetcapture2 interface=eth0 file=test.pcap timeout=1 "tcp and host 10.10.10.10 and port 443"
!pingPings the specified host.!ping 10.36.11.240

!ping type=proxy 10.36.11.240
!restartRestarts the specified Collector service!restart watchdog

!restart collector
!shealthcheck func=<function> collector=<collector_id>!shealthcheck commands help you determine the health of Collector, memory consumed, and number of scheduled tasks. Based on the result, you can debug the issues and resolve them.

func=trigger
This debug command triggers a healthcheck task for a specific Collector. After you run this command, Santaba schedules a healthcheck task for the specified Collector.
To view the result, you must run !shealthcheck command with function func=detail or func=show.

func=show
Typically, you will find a summary of the scheduled task in the result. It also indicates the number of total, finished, and skipped healthcheck tasks. For example, “Latest run has finished. Scheduled checks (total=34, finished=27, skipped=7)”
In case an issue is detected, the issue is summarised in the result. For example, “The disk has a low free space 728.57 MBytes.”
You can run !shealthcheck func=detail collector=<collector_id> debug command to know the details of the issue or get details of the health status of a Collector.

func=detail
This debug command provides status of all the scripts in detail. For example, “Collector exported jars and executables are not modified.” “The Collector has 177 instances.” 
After you run this command, Collector displays the result fetched by the func=trigger command. For example, for a specific Collector if you run the func=trigger command at 9.00 AM and then run func=detail at 11.00 AM, the Collector health status fetched at 9.00 AM will be displayed.		!shealthcheck func=show collector=123

!shealthcheck func=trigger collector=123

!shealthcheck func=detail collector=123
!tdetail <taskId>Displays detailed information about a specific data collection task, where taskId can be found with !tlist.!tdetail 12323209239991
!tlistLists the Collector’s data collection tasks, including DataSources, ConfigSources, and EventSources. A taskID is returned for each task.

You can narrow the list using the h=<hostname> and c=<collection type> options. 		!tlist c=wmi
!tlist summary=collector
!tlist summary=true lasttime=10 columns=5

Narrow the returned list:
!tlist c=script
!uptimeDisplays the uptime of the Collector.!uptime

In addition, you can enter the following if you need the syntax for a particular command or usage details (for example, optional and mandatory arguments or parameters) and examples, replacing commandname with the applicable command:

help !<commandname>

Requirements for Running the Collector Debug Facility

When running commands using the Collector Debug Facility, the command must be preceded with the following syntax:

!

Running the Collector Debug Facility

  1. Do one of the following depending on where you want to run the Collector Debug Facility from:
    • To run the Collector Debug Facility from the Collector’s settings, do the following:
      1. Navigate to Settings > Collectors.
      2. Select Manage for the Collector you want to debug from the list of Collectors.
        The Collector’s settings display in a panel.
        Collectors list
      3. Select More more icon in the settings panel, and then select “Run Debug Command”.
      4. Select Confirm when prompted.
        Debug terminal
        The Collector Debug Facility opens in a new tab.
    • To run the Collector Debug Facility from the DataSource in Resources, do the following:
      1. Navigate to Resources > Resources Tree, and select the DataSource or Datasource instance you want to debug.
      2. On the Raw Data tab, select Open Debug Window .
        open debug terminal icon
        The Collector Debug Facility opens in a new tab.
  2. Enter the applicable commands in the Collector Debug Facility.
    For example, use the !tlist command to list the Collector’s data collection tasks, including DataSources, ConfigSources, and EventSources.
    For more information, see Debug Commands for the Collector Debug Facility.

Recommendations:

  1. Close the window when finished.

You can restart a collector from the LogicMonitor platform or from the collector host. When the collector is up and running, you can restart the collector from the LogicMonitor platform. If the collector is down or dead, you have to restart it from the collector host.

Restarting from LogicMonitor Portal

To restart a collector from the LogicMonitor platform:

  1. Navigate to Settings > Collectors.
  2. Under the Collectors tab, select the collector that you want to restart.  
  3. Select the More option and then select Restart Collector.
    restart collector option
    A message confirming the restart is displayed.
  4. Select Confirm. The collector restart begins.

Restarting from Collector Host

To restart a collector on a Windows host, use the Services control panel to restart the following services:

To restart a collector on a Linux host, run the following commands:

To restart a collector from the LogicMonitor platform:

  1. Stop LogicMonitor: /usr/local/logicmonitor/agent/bin/sbshutdown
  2. Start the Watchdog service which may be run from init.d or systemd.
    • From init.d – /etc/init.d/logicmonitor-watchdog start
    • From systemd – systemctl start logicmonitor-watchdog

We’ve compiled some helpful tips for troubleshooting common Linux Collector issues.

Name Service Caching Daemon (NSCD)

The LogicMonitor Collector makes DNS queries to resolve the hosts it is monitoring and to determine which LogicMonitor servers to report data to.

If you are running an NSCD, you should make sure that it respects positive DNS TTLs. For example, on RedHat ES/CentOS glibc-2.5-24 and earlier does not respect DNS TTLs correctly, meaning that you will need to run a later version or disable NSCD.

If the NSCD returns a stale record for an hour, it would impact your ability to monitor devices and communication between LogicMonitor datacenters.

SE Linux

Some distributions of Linux (such as RedHat, CentOS, and Fedora) may have Security-Enhanced (SE) Linux enabled, which may restrict access and permissions that can impact the Collector’s ability to monitor.

Use the /usr/sbin/getenforce or /usr/sbin/sestatus commands to check the status of SELinux.

If you are having monitoring problems in your Linux server and SE Linux is enabled, you may be able to use Permissive mode to review the logs and identify the issue:

setenforce Permissive

In Permissive mode, more denials are logged because actions that would otherwise be denied in Enforcing mode are allowed.  After you run the Collector services in permissive mode and SE Linux logs to identify the permissions that need to be enabled, you should enable the necessary permissions.

To put SE Linux back into enforcing mode, run:

setenforce 1

We’ve compiled some helpful tips for troubleshooting common Windows Collector issues.

Error 1069

A common error encountered for Windows Collectors:

Error 1069: The service did not start due to a logon failure.

The account used to run the LogicMonitor Collector service on Windows must have “Log on as a service” rights on the host machine’s local security policy.

To update the local security policy:

1. Log on to the Collector host as a Local Administrator.

2. From CMD, PowerShell, or Run launch secpol.msc.

3. Expand “Local Policy” and click “User Rights Assignment”.

4. Right-click “Log on as a service” and select “Properties”.

5. Click “Add User or Group” to add the account which is to run the LogicMonitor Collector services.

6. In the “Select Users or Groups” dialog, find the user account and click “OK”.

7. In the “Log on as a service” properties window, click “OK” to save your changes.

If you are not able to make these changes (for example options may be greyed out or unavailable), you may need to look into what is enforcing the local security policy settings on the host–it may be the local or domain Group Policy settings or another configuration management tool.

Windows Monitoring Scenarios

Common monitoring scenarios on Windows use WMI and Perfmon. If you have issues related to monitoring WMI or Perfmon, refer to the following resources:

When you need to restart a Collector, you can do so from within LogicMonitor or from the Collector host.

Note: You can only use LogicMonitor to restart the Collector while it is up and running. If the Collector is down or dead, you will need to restart it from the Collector host.

Restart from LogicMonitor

To remotely restart a Collector from within the LogicMonitor platform:

1. Navigate to Settings | Collectors.

2. In the table, ind the Collector and click its Manage icon.

3. In the Manage Collector dialog, click Support and select “Restart Collector” from the menu.

Restarting a Collector from within the LogicMonitor platform

Restart from Collector Host

On a Windows host. To restart a Collector, use the Services control panel to restart the following services:

On a Linux host. To restart a Collector, run the following commands:

1. Stop LogicMonitor: /usr/local/logicmonitor/agent/bin/sbshutdown

2. Then start the watchdog service, which may be run from init.d or systemd.

Overview

You can use the Collector Debug Facility to remotely run debug commands on your Collector. This is helpful for troubleshooting issues with data collection and is typically used on the advice of LogicMonitor support.

Note: The history of Collector debug commands is preserved in the Audit Log.

Accessing the Collector Debug Facility

There are two places from which you can launch the Collector Debug Facility:

Debug Command Syntax

The Collector Debug Facility launches in a new browser tab. A list of built-in debug commands and their descriptions display to assist with troubleshooting.

All debug commands should be preceded with a ‘!’. If you need syntax for a particular command, enter help !<commandname>, as shown next.

Debug command syntax

The following table highlights some of the most frequently used debug commands. For usage details (e.g. optional and mandatory arguments, parameters, etc.) and examples, enter the help !<commandname> from the Collector Debug Facility.

CommandDescriptionExample
!accountDisplays the account information used by sbwinproxy.!account
!adlistDisplays a list of the Collector’s Active Discovery tasks. A taskID is returned for each task.!adlist type=get
!adlist method=ad_snmp
!adetail <taskId>Displays detailed information about a specific Active Discovery task, where taskId can be found with !adlist. Note that the “taskId” reference in the command specification will be labeled as “id” in the output of the !adlist command.!adetail 142
!checkcredentialEnables, disables, and checks credential usage on the specified host to determine source of unexpected login action.!checkcrendential proto=snmp user=public
!checkcrendential proto=snmp user=public usage=AP
!hostpropertyAdds, updates, or deletes system properties for a host.!hostproperty action=del host=localhost property=virtualization

 

!hostproperty action=add host=localhost property=ips value=127.0.0.1,192.168.1.1

!httpSends an HTTP request (with optional username and password) and displays the response.!http http://www.google.com/index.html
!jdbcExecutes a SQL query against the specified host.!jdbc ‘url=jdbc:mysql://productrds.chqwqvn285rf.us-west-2.rds.amazonaws.com:3306 username=LogicMonitor password=MyPassword’ select Name, ID from productDB.Employees
!logsurfDisplays log file entries that are of the specified debug level. If included, logs will only be displayed for the specified seq and taskId if they are in the specified file, and only n number of logs will be displayed. taskId and seq can be found using !tlist, where taskId is the id of a data collection task and seq is the number of times the collector remembers having done the task.!logsurf level=trace ../logs/wrapper.log taskid=833 seq=75
!pingPings the specified host.!ping 10.36.11.240
!ping type=proxy 10.36.11.240
!restartRestarts the specified Collector service!restart watchdog
!restart collector
!shealthcheck func=<function> collector=<collector_id>!shealthcheck commands help you determine the health of Collector, memory consumed, number of scheduled tasks, and more. Based on the result, you can debug the issues and resolve them.

func=trigger
This debug command triggers a healthcheck task for a specific Collector. Once you run this command, Santaba will schedule a healthcheck task for the specified Collector.
To view the result, you must run !shealthcheck command with function func=detail or func=show.

func=show
Typically, you will find a summary of the scheduled task in the result. It also indicates the number of total, finished, and skipped healthcheck tasks. For example, “Latest run has finished. Scheduled checks (total=34, finished=27, skipped=7)”
In case an issue is detected, the issue is summarised in the result. For example, “The disk has a low free space 728.57 MBytes.”
You can run !shealthcheck func=detail collector=<collector_id> debug command to know the details of the issue or get details of the health status of a Collector.

func=detail
This debug command provides status of all the scripts in detail. For example, “Collector exported jars and executables are not modified.” “The Collector has 177 instances.” 
Once you run this command, Collector will display the result fetched by the func=trigger command. For example, for a specific Collector if you run the func=trigger command at 9.00 AM and then run func=detail at 11.00 AM, the Collector health status fetched at 9.00 AM will be displayed.		!shealthcheck func=show collector=123
!shealthcheck func=trigger collector=123
!shealthcheck func=detail collector=123
!tdetail <taskId>Displays detailed information about a specific data collection task, where taskId can be found with !tlist.!tdetail 12323209239991
!tlistLists the Collector’s data collection tasks, including DataSources, ConfigSources, and EventSources. A taskID is returned for each task.!tlist c=wmi
!tlist summary=collector
!tlist summary=true lasttime=10 columns=5
!uptimeDisplays the uptime of the Collector.!uptime

Debug Example: Troubleshooting Data Collection

One of the most common uses of the Collector Debug Facility is troubleshooting data collection for a particular DataSource or DataSource instance. Maybe you just wrote a script DataSource and are getting NaN values, or perhaps one instance out of ten is not reporting data. You can typically use the following steps to identify the issue:

  1. Identify the DataSource or DataSource instance. Find the name of the DataSource in the DataSource definition (this is NOT the same as the display name).
    Example : troubleshooting data collection

  2. Use the !tlist command in the Collector Debug Facility of the collector associated with the device the DataSource applies to. You can narrow down the results by using the h=<hostname> and c=<collection type> options.

  3. Identify the task for the desired DataSource. You’ll see the taskid, followed by an execution count, followed by the collector type, a status, the device name, then the DataSource name, an ival (which is the amount of time it took to execute the task the last time), and finally, a note about the execution.
  4. Use the !tdetail command with the taskid as the argument.

  5. If you need more information to diagnose the problem, increase the log level for the appropriate collection method of the Collector, as discussed in Collector Logging.
  6. Wait a polling cycle (or more) and then use the !logsurf command with taskid as an argument and ../logs/wrapper.log as the filename (if you know the latest execution count, you can also limit the results to one operation by including seq=n). You can also include a number argument to limit the results to a certain number of logs. You should see the log entries only for the task whose id is included in the command.

  7. If you still haven’t identified the issue, contact support.