Collecting and Forwarding Syslog Logs

Syslog (syslog, rsyslog, syslog-ng) is one of the most common sources of log data in enterprise environments. The LogicMonitor Collector has the capability to receive Syslog data and forward the raw logs to the LM Logs Ingestion API. 

Prerequisites

• EA Collector 29.104 or later installed.
• Syslog collection and log forwarding requires port 514/UDP to be open and available on both the Collector machine and firewall. (Note: This may require disabling UDP for rsyslog.)

Enable Syslog Log Forwarding

As with previous versions, the Collector can be configured to receive syslog messages by listening on port 514/UDP:

• Devices monitored by the Collector can forward syslog messages from an existing centralized syslog server.
• Monitored devices that have a syslog client can be configured to send the syslog data to the Collector.
• Monitored devices are not required to send syslog messages to the same collector that they are monitored by.

To turn on Syslog forwarding for LM Logs, uncomment the following property in the Collector’s agent.conf. If the property doesn’t exist, you can add it:

# Enable LM Logs
lmlogs.syslog.enabled=true

Configure Resource Mapping

Once Syslog events are received, the Collector will map them to existing monitored resources before forwarding them on to the logs ingestion API. If the log cannot be mapped to an LM-monitored resource, you can still view and and search the log by “_resource.id=0”.

For the syslog logs sent from the Collector, uncomment the following properties in agent.conf for device mapping:

If the value of system.hostname for your monitored resources does not match their IPs reported in the Syslog events, then these default settings will not work and device mapping will fail. In this case, you will need to change the lmlogs.syslog.property.name to a property which matches the value of the lmlogs.syslog.hostname.format setting.

If you do not have an existing LogicMonitor property matching any of these values, you can create a PropertySource to add the attribute across your environment or set the property manually.

Note: If you are forwarding syslog logs from Cisco devices, the LogicMonitor Collector will attempt to use the socket address if the IP or hostname are not available. For this to work, the Collector receiving the logs needs to be monitoring the Cisco devices sending the logs.

Configure Filters to Remove Logs

We recommend that you configure filters to remove log messages that contain sensitive information (such as credit cards, phone numbers, or personal identifiers) so that they are not sent to LogicMonitor. Filters can also be used to reduce the volume of non-essential syslog log messages that are sent to the logs ingestion API queue.

The filtering criteria are based on the syslog fields: severity, message, facility, and application. To configure a filter criteria, uncomment to add or edit the following entries in agent.conf:

Date and Timestamp Parsing

If the date and time format in the logs is not supported or the time information is missing, LogicMonitor may not be able to parse the logs properly. When this happens you can either specify a custom date and time format or use the timestamp of the event when it’s received. These settings can be specified in the Collector’s agent.conf.

For example, if you have logs with the following date format (such as from Palo Alto Networks Panorama devices): [2020-09-18 10:03:58.434 UTC]; you may need to add the custom date format below:

eventcollector.syslog.supportDateFormat.8=yyyy-MM-dd'T'HH:mm:SSxxx

If you have syslog events that do not include time information, enable the following setting to use the timestamp when the log was received by the Collector:

lmlogs.syslog.useTimestampWhenCollectorReceivedLogs=TRUE

Troubleshooting

If you’ve enabled syslog log forwarding on the Collector, but the logs are not displaying in the Logs Page, you can perform the following checks to troubleshoot the issue.

Collector Configuration

Note: Collector configuration changes are universal on the Collector and can affect every device sending logs to that Collector. Changing a flag may fix one device but could affect others. It is usually a combination of changes that need to be done to resolve all syslog device issues.

In the Collector’s configuration file (agent.conf):

1. Is LM Logs enabled?

Check that lmlogs.syslog.enabled=true and that the values for lmlogs.syslog.property.name and lmlogs.syslog.hostname.format match.

Note: lmlogs.syslog.property.name can be set to ANY property in LogicMonitor. The default is system.hostname, but system.sysname or displayname can also help if naming and device configurations are inconsistent.

2. Are any filters defined that may be responsible for filtering out the logs you expect to see?

3. Were there mapping errors?

Check the Collector events to identify mapping errors.

• Change lmlogs.syslog.property.name and lmlogs.syslog.hostname.format appropriately to make sure that the right information from the syslog event is mapped against the LogicMonitor property value.
• Use lmlogs.syslog.hostname.format=DO_NOTHING to bypass DNS if hostnames are already resolved. This is not listed in the configuration file by default.
• Some resource types do not pass along the hostname with syslog metadata, but the hostname is in the message field. Set lmlogs.syslog.UseHostNameFromMessages=true to use the hostname from the message.
• If the hostname is not valid, LogicMonitor defaults to using the socket address for device mapping. Check that lmlogs.syslog.UseSocketAddressIfhostNameisInvalid=true.

4. Are the Logs in a timezone >= 3 hours from the Collector?

LM Logs will not accept logs that are more than 3 hours from the current. To avoid this issue, you can set the timestamp to the time the Collector receives the log events with lmlogs.syslog.useTimestampWhenCollectorReceivedLogs=true.

5. Check the Collector version and sizing.

Refer to the Collector Capacity for syslog deployments. If needed, implement a standalone LM Logs Collector which is not managing any other resources.

6. Check Collector Events.

In Manage Collector, expand the Support dropdown and select Collector Events. You can review mapping errors and search using keywords.

7. Check the Collector LM Logs graphs and Metrics.

Expand the Collector Resource > Collector DataSource Group > Collector LM Logs DataSource.

• If queuing issues are happening and debug logs also show an increasing queue size, adjust the size for lmlogs.thread.count.for.ingest.api.communication=10. We recommend that you double the default size (which is 10) and view the results in the graphs before increasing the size further.
• Other graphs include the number of incoming events, logs dropped, mapping errors, events ignored due to filters, and more.

Debug Logs

1. Adjust the logging level on the Collector to debug and review wrapper.log.

• In Settings > Collector > Logs > Manage, set the logging level for he eventcollector.syslog component to debug. (The default is info.)
• In Manager Collector, expand the Support dropdown and select “Send logs to LogicMonitor”. Then, review the wrapper.log files with the Log tab for the given collector.

Note: Don’t forget to change the logging level back to info when you’re done troubleshooting.

2. Live tail wrapper logs.

• In Manage Collector, expand the Support dropdown and select “Run Debug Command”.
• Enter the following to show the last 500 lines in wrapper.log:  !tail ..\logs\wrapper.log 500 syslog You may also change syslog to an IP address or grep another keyword.