Collecting and Forwarding Syslog Logs

Syslog (syslog, rsyslog, syslog-ng) is one of the most common sources of log data in enterprise environments. Beginning with version EA 29.104, the LogicMonitor Collector has the capability to receive Syslog data and forward the raw logs to the LM Logs Ingestion API. If you are already using the LogicMonitor Collector, this is distinct from the Syslog EventSource configuration for alerting on Syslog metrics.

Note: The Collector can be used for alerting on Syslog metrics or act as a Syslog Log Collector for LM Logs. It cannot do both.

Prerequisites

• Syslog collection and log forwarding requires port 514/UDP to be open and available on both the Collector machine and firewall. (Note: This may require disabling UDP for rsyslog.)

Enable Syslog log forwarding

As with previous versions, the Collector can be configured to receive syslog messages by listening on port 514/UDP:

• Devices monitored by the Collector can forward syslog messages from an existing centralized syslog server.
• Monitored devices that have a syslog client can be configured to send the syslog data to the Collector.
• Monitored devices are not required to send syslog messages to the same collector that they are monitored by.

To turn on Syslog forwarding for LM Logs, uncomment the following property in the Collector’s agent.conf. If the property doesn’t exist, you can add it:

# Enable LM Logs
lmlogs.syslog.enabled=true

Configure resource mapping

Once Syslog events are received, the Collector will map them to existing monitored resources before forwarding them on to the logs ingestion API. The Collector will not forward syslog messages that cannot be associated with existing monitored resources.

For the syslog logs sent from the Collector, uncomment the following properties in agent.conf to use the system.hostname property for device mapping:

# Resource properties which we want to use for device mapping
lmlogs.syslog.property.name=system.hostname

# Resource properties value format. Possible values are IP, FQDN, HOSTNAME, DO_NOTHING
lmlogs.syslog.hostname.format=IP

If the value of system.hostname for your monitored resources does not match their IPs reported in the Syslog events, then these default settings will not work and device mapping will fail. In this case, you will need to change the lmlogs.syslog.property.name to a property which matches the value of the lmlogs.syslog.hostname.format setting.

The Collector will bypass DNS resolution for syslog messages that include the hostname in the header if lmlogs.syslog.hostname.format=DO_NOTHING. This setting will avoid attempts to resolve the hostname when it is already known.

If you do not have an existing LogicMonitor property matching any of these values, you can create a PropertySource to add the attribute across your environment or set the property manually.

Configure filters to remove logs

We recommend that you configure filters to remove log messages that contain sensitive information (such as credit cards, phone numbers, or personal identifiers) so that they are not sent to LogicMonitor. Filters can also be used to reduce the volume of non-essential syslog log messages that are sent to the logs ingestion API queue.

The filtering criteria are based on the syslog fields: severity, message, and facility. To configure a filter criteria, add the following to agent.conf:

logsource.syslog.filter.<serial number of filter>.<filter name>.<operator>=<value>

See examples below:

logsource.syslog.filter.1.severity.equals=informational 
logsource.syslog.filter.2.severity.moreUrgentThan=warning
logsource.syslog.filter.3.facility.equals=user-level messages
logsource.syslog.filter.4.facility.notEqual=mail system
logsource.syslog.filter.5.message.equal=<message>
logsource.syslog.filter.6.message.notEqual=<message>
logsource.syslog.filter.7.message.contain=<message>
logsource.syslog.filter.8.message.notContain=<message>
logsource.syslog.filter.9.message.regexMatch=<pattern>
logsource.syslog.filter.10.message.regexNotMatch=<pattern>

Date and timestamp parsing

If the date and time format in the logs is not supported or the time information is missing, LogicMonitor may not be able to parse the logs properly. When this happens you can either specify a custom date and time format or use the timestamp of the event when it’s received. These settings can be specified in the Collector’s agent.conf.

For example, if you have logs with the following date format (such as from Palo Alto Networks Panorama devices): [2020-09-18 10:03:58.434 UTC]; you may need to add the custom date format below:

eventcollector.syslog.supportDateFormat.8=yyyy-MM-dd'T'HH:mm:SSxxx

If you have syslog events that do not include time information, enable the following setting to use the timestamp when the log was received by the Collector:

lmlogs.syslog.useTimestampWhenCollectorReceivedLogs=TRUE

Troubleshooting

If you’ve enabled syslog log forwarding on the Collector, but the logs are not displaying in the Logs Page, you can perform the checks for resource mapping, hostname retrieval, and log ingestion.

Enable debug logging

Make sure that debug logging is enabled on the LogicMonitor Collector to get detailed information about LM Logs:

1. In Settings | Collector, click the Logs icon for the selected collector.

2. Select Manage to see the logging levels for each component for that collector.

3. For the eventcollector.syslog component, select debug to enable debug logging for syslog logs.

For more information, see Collector Logging.

Resource mapping errors

When resource mapping fails, the Collector will ignore the syslog messages from these devices and will not forward the logs to the ingestion endpoint until a configurable time interval has elapsed.

This time interval can be configured in agent.conf and defaults to 60 mins:

lmlogs.syslog.ignored.resource.cache.time.min=60

If this doesn’t help, you can check for the following messages in the Collector’s wrapper.log:

• Added the rejected resource ID in Collector cache
• Host id is there in the rejected list since device mapping failed. Ignoring the messages

The Collector’s wrapper.log may also return the following response codes from the log ingestion API:

• 207 Multi-status
• 400 Bad Request

DNS resolution failure

DNS name resolution for the hostname received in the syslog message header will be performed by the Collector.

If the Collector fails to retrieve the correct hostname:

• Check the values of the following agent.conf properties: lmlogs.syslog.property.name and lmlogs.syslog.hostname.format
• Check wrapper.log for the following message: Host is not present so ignoring this message

Log ingestion issues

If device mapping and hostname retrieval are correct, you may want to check that the logs are being ingested.

To do this, search the Collector logs for the Info message: Response received from Ingest API. This message will include the response code and request ID from the ingestion endpoint.

For more information, see Log Ingestion API Response Codes.