Sending Syslog Logs

Syslog (syslog, rsyslog, syslog-ng) is one of the most common sources of log data in enterprise environments. Syslog data originates in the infrastructure and is forwarded to a target IP address which for LogicMonitor is the Collector. The Collector currently monitoring the resource does not have to be same one that receives the Syslog messages.

Requirements

• EA Collector 29.104 or later installed.
• Syslog collection and log forwarding requires port 514/UDP to be open and available on both the Collector machine and firewall. Note: This may require disabling UDP for rsyslog.

Enabling Syslog Log Forwarding

The LM Collector receives Syslog data and forwards the raw logs to the LM Logs Ingestion API for viewing, searching and analysis.

As with previous versions, the Collector can be configured to receive syslog messages by listening on port 514/UDP:

• Devices monitored by the Collector can forward syslog messages from an existing centralized syslog server.
• Monitored devices that have a syslog client can be configured to send the syslog data to the Collector.
• Monitored devices are not required to send syslog messages to the same Collector that they are monitored by.

To turn on Syslog forwarding for LM Logs, uncomment the following property in the Collector’s agent.conf. If the property doesn’t exist, you can add it:

# Enable LM Logs
lmlogs.syslog.enabled=true

Configuring Resource Mapping

Once Syslog events are received, the Collector will map them to existing monitored resources before forwarding them on to the logs ingestion API. If the log cannot be mapped to an LM-monitored resource, you can still view and and search the log by “_resource.id=0”.

For the syslog logs sent from the Collector, uncomment the following properties in agent.conf for device mapping:

If the value of system.hostname for your monitored resources does not match their IPs reported in the Syslog events, then these default settings will not work and device mapping will fail. In this case, you will need to change the lmlogs.syslog.property.name to a property which matches the value of the lmlogs.syslog.hostname.format setting.

If you do not have an existing LogicMonitor property matching any of these values, you can create a PropertySource to add the attribute across your environment or set the property manually.

Configuring Filters to Remove Logs

We recommend that you configure filters to remove log messages that contain sensitive information (such as credit cards, phone numbers, or personal identifiers) so that they are not sent to LogicMonitor. Filters can also be used to reduce the volume of non-essential syslog log messages that are sent to the logs ingestion API queue.

The filtering criteria are based on the syslog fields: severity, message, facility, and application.

To configure a filter criteria, uncomment to add or edit the following entries in agent.conf:

Date and Timestamp Parsing

If the date and time format in the logs is not supported or the time information is missing, LogicMonitor may not be able to parse the logs properly. When this happens you can either specify a custom date and time format or use the timestamp of the event when it’s received. These settings can be specified in the Collector’s agent.conf.

For example, if you have logs with the following date format (such as from Palo Alto Networks Panorama devices): [2020-09-18 10:03:58.434 UTC]; you may need to add the custom date format below:

eventcollector.syslog.supportDateFormat.8=yyyy-MM-dd'T'HH:mm:SSxxx

If you have syslog events that do not include time information, enable the following setting to use the timestamp when the log was received by the Collector:

lmlogs.syslog.useTimestampWhenCollectorReceivedLogs=TRUE

Troubleshooting

If you’ve enabled syslog log forwarding on the Collector, but the logs are not displaying in the Logs page, you can perform the following checks to troubleshoot the issue.

Collector Configuration

Check the following in the Collector’s configuration file (agent.conf):

1. Is LM Logs enabled?

Check that lmlogs.syslog.enabled=true and that the values for lmlogs.syslog.property.name and lmlogs.syslog.hostname.format match.

lmlogs.syslog.property.name can be set to ANY property in LogicMonitor. The default is system.hostname, but system.sysname or displayname can also help if naming and device configurations are inconsistent.

2. Are any filters defined that may be responsible for filtering out the logs you expect to see?

3. Were there mapping errors?

Check the Collector events to identify mapping errors.

• Change lmlogs.syslog.property.name and lmlogs.syslog.hostname.format appropriately to make sure that the right information from the syslog event is mapped against the LogicMonitor property value.
• Use lmlogs.syslog.hostname.format=DO_NOTHING to bypass DNS if hostnames are already resolved. This is not listed in the configuration file by default.
• Some resource types do not pass along the hostname with syslog metadata, but the hostname is in the message field. Set lmlogs.syslog.UseHostNameFromMessages=true to use the hostname from the message.
• If the hostname is not valid, LogicMonitor defaults to using the socket address for device mapping. Check that lmlogs.syslog.UseSocketAddressIfhostNameisInvalid=true.

4. Are the Logs in a timezone >= 3 hours from the Collector?

LM Logs will not accept logs that are more than 3 hours from the current. To avoid this issue, you can set the timestamp to the time the Collector receives the log events with lmlogs.syslog.useTimestampWhenCollectorReceivedLogs=true.

5. Check the Collector version and sizing.

Refer to the Collector Capacity for syslog deployments. If needed, implement a standalone LM Logs Collector which is not managing any other resources.

6. Check Collector Events.

In Manage Collector, expand the Support dropdown and select Collector Events. You can review mapping errors and search using keywords.

7. Check the Collector LM Logs graphs and Metrics.

Expand the Collector Resource > Collector DataSource Group > Collector LM Logs DataSource.

• If queuing issues are happening and debug logs also show an increasing queue size, adjust the size for lmlogs.thread.count.for.ingest.api.communication=10. We recommend that you double the default size (which is 10) and view the results in the graphs before increasing the size further.
• Other graphs include the number of incoming events, logs dropped, mapping errors, events ignored due to filters, and more.

Debug Logs

1. Adjust the logging level on the Collector to debug and review wrapper.log.

• In Settings > Collector > Logs > Manage, set the logging level for the eventcollector.syslog component to debug. (The default is info.)
• In Manager Collector, expand the Support dropdown and select “Send logs to LogicMonitor”. Then, review the wrapper.log files with the Log tab for the given collector.

2. Live tail wrapper logs.

• In Manage Collector, expand the Support dropdown and select “Run Debug Command”.
• Enter the following to show the last 500 lines in wrapper.log:  !tail ..\logs\wrapper.log 500 syslog You may also change syslog to an IP address or grep another keyword.

UDP Port 514 and Firewall issues for Syslogs on Windows Collector

Due to inbound firewall rules, Syslogs may not reach the Windows collector which listens on port 514. In such a case, you can troubleshoot the issue using the following steps.

1. Check if the firewall is enabled.

In PowerShell, run the following command to check if the firewall is enabled.

$firewallStatus = Get-NetFirewallProfile | Select-Object -ExpandProperty Enabled; if($firewallStatus -eq "True") { Write-Host "Windows Firewall is Enabled" } else { Write-Host "Windows Firewall is Disabled" }

• If the Windows firewall is enabled, check if the inbound firewall rules are added to allow the Syslogs from remote hosts to reach the Windows machine.
• If the inbound rules are not added, add them for the respective remote hosts.

Alternatively, to troubleshoot the issue, you can temporarily disable the Windows firewall.

2. Check if any security software or antivirus software is running on the Windows machine.

Your Windows machine may be running security or antivirus software configured to reject traffic on port 514. In PowerShell, to check all the processes on the Windows machine, run the following command:

Get-Process | Select-Object Name

The output displays the name of the processes running on the Windows machine. If any security software or antivirus software is running, you can change their configuration to allow traffic on port 514 from the intended IPs.

3. Check if any security software or antivirus software is running on the Windows machine.
The Windows collector cannot access port 514 if the port is already used by another application or service. In PowerShell, run the following command to check the process ID which is accessing the Syslogs on port 514.

netstat -ano | findstr 514

If there is another application or service, stop or kill that process and restart the collector.