Troubleshooting Syslog Log Ingestion

If syslog log forwarding has been enabled on the Collector but logs are not displaying on the Logs page, you can perform the following steps to troubleshoot the issue.

Log Ingestion Issues when using the Collector

There may be issues with how the Collector is configured, resulting in missing or delayed logs, unmapped (deviceless) logs, or partial/incomplete log messages. Common examples include the following:

• The LM Logs tab shows zero log entries for expected time windows
• Tracked queries or log volume dashboards drop to zero
• Partial log messages display in LM Logs
• Logs are collected but show up as “unmapped” or “deviceless” logs
• Ingestion stalls or becomes erratic
• Collector metrics show high CPU/memory or log ingestion backlog
• Logs fail to parse or log field extraction fails
• Unexpected or null values in parsed fields like event.code, hostname, etc
• Logs arrive minutes after the actual event time

To troubleshoot, you can check filters, mapping errors, time zones, Collector version and sizing, LM Logs graphs and metrics, the wrapper.log file, and if LM Logs is enabled.

Verifying LM Logs is Enabled on the Collector

If you are using the legacy method to configure syslog ingestion using the agent.conf file on a given Collector, check that the agent.conf file contains lmlogs.syslog.enabled=true and that the values for lmlogs.syslog.property.name and lmlogs.syslog.hostname.format match. The lmlogs.syslog.property.name parameter can be set to any property in LogicMonitor. The default parameter is system.hostname, but you can use system.sysname or system.displayname if the naming and device configurations are inconsistent.

To review the agent.conf file, see Editing the Collector Configuration Profile.

Checking Filters

If you are using the legacy method to configure syslog ingestion using the agent.conf file on a given Collector, verify that the filters you applied are not filtering out logs you expect to see. To review your syslog log filters, navigate to My Module Toolbox, select the module that is ingesting syslogs, and select the Filters tab. For more information about filters, see Configuring Filters to Remove Logs. 

Checking for Mapping Errors

If you are ingesting syslog logs using a syslog LogSource, see Resource Mappings.

If you are ingesting syslog logs by modifying the Collector file, do the following:

• Change the lmlogs.syslog.property.name and lmlogs.syslog.hostname.format parameters to ensure information from the syslog event is mapped against the LogicMonitor property value.
• Insert lmlogs.syslog.hostname.format=DO_NOTHING into the agent.conf file to bypass DNS if hostnames are already resolved. This is not listed in the configuration file by default.
• Some resource types do not pass the hostname with syslog metadata, but the hostname is in the message field. Insert lmlogs.syslog.UseHostNameFromMessages=true into the agent.conf file to use the hostname from the message.
• If the hostname is not valid, LogicMonitor defaults to using the socket address for device mapping. Check that lmlogs.syslog.UseSocketAddressIfhostNameisInvalid=true is in the agent.conf file. 

Verifying Time Zones

LM Logs will not accept logs that are recorded in a timezone that differs by more than 3 hours from the Collector time zone. To avoid this issue, you can set the timestamp to the time the Collector receives the log events in one of the following ways:

• If you are ingesting syslog logs using a syslog LogSource, toggle the Use received time instead of logstamp switch in the LogSource configuration.
• If you are ingesting syslog logs by modifying a Collector config file, insert lmlogs.syslog.useTimestampWhenCollectorReceivedLogs=true into the agent.conf file.

Checking the Collector Version and Sizing

Review the amount of data, load, and capacity that the Collector can handle. For details, see Collector Capacity. If needed, implement a standalone LM Logs Collector that is not managing any other resources.

Checking the Collector LM Logs Graphs and Metrics

In LogicMonitor, navigate to the Resources page and identify the Collector you are forwarding logs to. Expand the Collector Resource > Collector DataSource Group > Collector LM Logs DataSource.

• If you encounter a growing queue from the Collector DataSource, adjust the size for lmlogs.thread.count.for.ingest.api.communication=10 in the agent.conf file. It is recommended to double the default size of 10 and view the results in the graphs before further increasing the size.
• You can also review other graphs that include the number of incoming events, logs dropped, mapping errors, events ignored due to filters, and more.

Reviewing the wrapper.log File using Debug Logs

Adjust the logging level on the Collector to Debug and review the wrapper.log file.

1. In LogicMonitor, navigate to Settings > Collector > Logs > Manage. Set the logging level for the eventcollector.syslog component to Debug. (The default logging level is Info.)
2. In Manager Collector, expand the Support dropdown and select Send logs to LogicMonitor. Review the wrapper.log files in the Log tab for the given Collector.
3. In Manage Collector, expand the Support dropdown and select Run Debug Command.
4. Enter the following to show the last 500 lines in the wrapper.log file:  !tail ..\logs\wrapper.log 500 syslog.

UDP Port 514 and Firewall Issues for Syslog Logs on Windows Collector

Due to inbound firewall rules, syslog logs may not reach the Collector that listens on port 514. You can check if logs are not reaching the Collector by reviewing the Windows Firewall, security software, and other processes listening on port 514.

Checking if the Windows Firewall is Enabled

In PowerShell, run the following command to check if the Windows Firewall is enabled.

$firewallStatus = Get-NetFirewallProfile | Select-Object -ExpandProperty Enabled; if($firewallStatus -eq "True") { Write-Host "Windows Firewall is Enabled" } else { Write-Host "Windows Firewall is Disabled" }

• If the Windows firewall is enabled, check if the inbound firewall rules are added to allow syslog logs from remote hosts to reach the Windows machine.
• If the inbound rules are not added, add them for the respective remote hosts.

Alternatively, you can temporarily disable the Windows firewall for troubleshooting purposes.

Checking if Security Software is Running

Your Windows machine may be running security or antivirus software configured to reject incoming traffic on port 514. In PowerShell, check all the processes on the Windows machine using the following command:

Get-Process | Select-Object Name

The output displays the name of the processes running on the Windows machine. If any security software or antivirus software is running, you can configure them to allow traffic on port 514 from the intended IP addresses.

You can verify whether syslog messages are reaching the Collector by performing a packet capture from the Collector host using one of the following common methods:

• On a Windows Collector, you can use Wireshark, a graphical network protocol analyzer that enables you to capture and analyze network traffic in real time. For details, see Capturing Live Network Data from Wireshark.
• On a Linux Collector, you can use tcpdump, a command-line packet analyzer that captures and displays network traffic in real time. For details, see An introduction to using tcpdump at the Linux command line from Red Hat.

Checking if Another Process is Listening on Port 514

The Windows and Linux Collectors cannot access port 514 if the port is already in use by another application or service.

If you are using a Windows Collector, run the following command in PowerShell to check the process ID that is accessing syslogs on port 514:

netstat -ano | findstr 514

If you are using a Linux Collector, the command to check the process in the Terminal will depend on which packages you have already installed. If you do not have NetTools installed, use the following command:

sudo ss -tulnp | grep ':514'

If you have NetTools installed, use the following command:

sudo netstat -tulnp | grep ':514'

If there is another application or service, stop that process and restart the Collector.