Syslog Log Ingestion Overview

Syslogs—including syslog, rsyslog, syslog-ng—are common sources of log data in enterprise environments. Syslog data originates from devices in the infrastructure and is forwarded to a target IP address. When using LM Logs, this target IP address is the LM Collector. The LM Collector that receives syslog logs does not need to be the same one monitoring the resource.

You can ingest syslog logs in one of two ways:

• Using the default LogSource—LogSources are the primary method for sending syslog to LM Logs. LogSources allow log filtering, custom metadata tagging, and reliable resource mapping. For more information, see LogSource Overview.
  Every LM Collector includes a pre-configured default LogSource for syslog ingestion. The default LogSource contains groups of settings for resource mappings and log fields that are parsed automatically during ingestion. These settings are provided out-of-the-box for a more streamlined user experience. For more information about resource mappings and log fields settings, see Syslog LogSource Configuration.
• Applying a LogSource to LM Collector—You can also ingest syslogs by applying a LogSource to the LM Collector.  This applies the LogSource configuration to all syslog sent to the LM Collector, regardless of resource. 

Requirements

• LM Collector installation—Syslog logs use the LM Collector. 

• Port 514 (UDP) — Port 514 (UDP) must be open and accessible on both the machine hosting the Collector and the firewall. This may require disabling UDP for rsyslog.

Troubleshooting

If you’ve enabled syslog log forwarding on the Collector, but the logs are not displaying in the Logs page, you can perform the following checks to troubleshoot the issue.

Collector Configuration

Check the following in the Collector’s configuration file (agent.conf):

1. Is LM Logs enabled?

Check that lmlogs.syslog.enabled=true and that the values for lmlogs.syslog.property.name and lmlogs.syslog.hostname.format match.

lmlogs.syslog.property.name can be set to ANY property in LogicMonitor. The default is system.hostname, but system.sysname or displayname can also help if naming and device configurations are inconsistent.

2. Are any filters defined that may be responsible for filtering out the logs you expect to see?

3. Were there mapping errors?

Check the Collector events to identify mapping errors.

• Change lmlogs.syslog.property.name and lmlogs.syslog.hostname.format appropriately to make sure that the right information from the syslog event is mapped against the LogicMonitor property value.
• Use lmlogs.syslog.hostname.format=DO_NOTHING to bypass DNS if hostnames are already resolved. This is not listed in the configuration file by default.
• Some resource types do not pass along the hostname with syslog metadata, but the hostname is in the message field. Set lmlogs.syslog.UseHostNameFromMessages=true to use the hostname from the message.
• If the hostname is not valid, LogicMonitor defaults to using the socket address for device mapping. Check that lmlogs.syslog.UseSocketAddressIfhostNameisInvalid=true.

4. Are the Logs in a timezone >= 3 hours from the Collector?

LM Logs will not accept logs that are more than 3 hours from the current. To avoid this issue, you can set the timestamp to the time the Collector receives the log events with lmlogs.syslog.useTimestampWhenCollectorReceivedLogs=true.

5. Check the Collector version and sizing.

Refer to the Collector Capacity for syslog deployments. If needed, implement a standalone LM Logs Collector which is not managing any other resources.

6. Check Collector Events.

In Manage Collector, expand the Support dropdown and select Collector Events. You can review mapping errors and search using keywords.

7. Check the Collector LM Logs graphs and Metrics.

Expand the Collector Resource > Collector DataSource Group > Collector LM Logs DataSource.

• If queuing issues are happening and debug logs also show an increasing queue size, adjust the size for lmlogs.thread.count.for.ingest.api.communication=10. We recommend that you double the default size (which is 10) and view the results in the graphs before increasing the size further.
• Other graphs include the number of incoming events, logs dropped, mapping errors, events ignored due to filters, and more.

Debug Logs

1. Adjust the logging level on the Collector to debug and review wrapper.log.

• In Settings > Collector > Logs > Manage, set the logging level for the eventcollector.syslog component to debug. (The default is info.)
• In Manager Collector, expand the Support dropdown and select “Send logs to LogicMonitor”. Then, review the wrapper.log files with the Log tab for the given collector.

2. Live tail wrapper logs.

• In Manage Collector, expand the Support dropdown and select “Run Debug Command”.
• Enter the following to show the last 500 lines in wrapper.log:  !tail ..\logs\wrapper.log 500 syslog You may also change syslog to an IP address or grep another keyword.

UDP Port 514 and Firewall issues for Syslogs on Windows Collector

Due to inbound firewall rules, Syslogs may not reach the Windows collector which listens on port 514. In such a case, you can troubleshoot the issue using the following steps.

1. Check if the firewall is enabled.

In PowerShell, run the following command to check if the firewall is enabled.

$firewallStatus = Get-NetFirewallProfile | Select-Object -ExpandProperty Enabled; if($firewallStatus -eq "True") { Write-Host "Windows Firewall is Enabled" } else { Write-Host "Windows Firewall is Disabled" }

• If the Windows firewall is enabled, check if the inbound firewall rules are added to allow the Syslogs from remote hosts to reach the Windows machine.
• If the inbound rules are not added, add them for the respective remote hosts.

Alternatively, to troubleshoot the issue, you can temporarily disable the Windows firewall.

2. Check if any security software or antivirus software is running on the Windows machine.

Your Windows machine may be running security or antivirus software configured to reject traffic on port 514. In PowerShell, to check all the processes on the Windows machine, run the following command:

Get-Process | Select-Object Name

The output displays the name of the processes running on the Windows machine. If any security software or antivirus software is running, you can change their configuration to allow traffic on port 514 from the intended IPs.

3. Check if any security software or antivirus software is running on the Windows machine.
The Windows collector cannot access port 514 if the port is already used by another application or service. In PowerShell, run the following command to check the process ID which is accessing the Syslogs on port 514.

netstat -ano | findstr 514

If there is another application or service, stop or kill that process and restart the collector.