Syslog (syslog, rsyslog, syslog-ng) is one of the most common sources of log data in enterprise environments. Beginning with version EA 29.104, the LogicMonitor Collector has the capability to receive Syslog data and forward the raw logs to the LM Logs Ingestion API. If you are already using the LogicMonitor Collector, this is distinct from the Syslog EventSource configuration for alerting on Syslog metrics.
Note: The Collector can be used for alerting on Syslog metrics or act as a Syslog Log Collector for LM Logs. It cannot do both.
- Syslog collection and log forwarding requires port 514/UDP to be open and available on both the Collector machine and firewall. (Note: This may require disabling UDP for rsyslog.)
Enable Syslog log forwarding
As with previous versions, the Collector can be configured to receive syslog messages by listening on port 514/UDP:
- Devices monitored by the Collector can forward syslog messages from an existing centralized syslog server.
- Monitored devices that have a syslog client can be configured to send the syslog data to the Collector.
- Monitored devices are not required to send syslog messages to the same collector that they are monitored by.
To turn on Syslog forwarding for LM Logs, uncomment the following property in the Collector’s
agent.conf. If the property doesn’t exist, you can add it:
# Enable LM Logs lmlogs.syslog.enabled=true
Configure resource mapping
Once Syslog events are received, the Collector will map them to existing monitored resources before forwarding them on to the logs ingestion API. The Collector will not forward syslog messages that cannot be associated with existing monitored resources.
For the syslog logs sent from the Collector, uncomment the following properties in
agent.conf to use the
system.hostname property for device mapping:
# Resource properties which we want to use for device mapping lmlogs.syslog.property.name=system.hostname # Resource properties value format. Possible values are IP,FQDN,HOSTNAME lmlogs.syslog.hostname.format=IP
If the value of
system.hostname for your monitored resources does not match their IPs reported in the Syslog events, then these default settings will not work and device mapping will fail. In this case, you will need to change the
lmlogs.syslog.property.name to a property which matches the value of the
If you do not have an existing LogicMonitor property matching any of these values, you can create a PropertySource to add the attribute across your environment or set the property manually.
Configure filters to remove logs
We recommend that you configure filters to remove log messages that contain sensitive information (such as credit cards, phone numbers, or personal identifiers) so that they are not sent to LogicMonitor. Filters can also be used to reduce the volume of non-essential syslog log messages that are sent to the logs ingestion API queue.
The filtering criteria are based on the syslog fields: severity, message, and facility. To configure a filter criteria, add the following to
logsource.syslog.filter.<serial number of filter>.<filter name>.<operator>=<value>
|Define filters for syslog messages with a severity level equal to or more urgent then the specified value.|
|Define filters for syslog messages based on the facility value: kernel messages, user-level messages, clock daemon, and so on.|
|Define filters for syslog messages based on the contents of the message itself, using keywords or a regular expression pattern to match.|
See examples below:
logsource.syslog.filter.1.severity.equals=info logsource.syslog.filter.2.severity.moreUrgentThan=warning logsource.syslog.filter.3.facility.equals=user-level messages logsource.syslog.filter.4.facility.notEqual=mail system logsource.syslog.filter.5.message.equal=<message> logsource.syslog.filter.6.message.notEqual=<message> logsource.syslog.filter.7.message.contain=<message> logsource.syslog.filter.8.message.notContain=<message> logsource.syslog.filter.9.message.regexMatch=<pattern> logsource.syslog.filter.10.message.regexNotMatch=<pattern>
If you’ve enabled syslog log forwarding on the Collector, but the logs are not displaying in the Logs Page, you can perform the checks for resource mapping, hostname retrieval, and log ingestion.
Enable debug logging
Make sure that debug logging is enabled on the LogicMonitor Collector to get detailed information about LM Logs:
1. In Settings | Collector, click the Logs icon for the selected collector.
2. Select Manage to see the logging levels for each component for that collector.
3. For the eventcollector.syslog component, select debug to enable debug logging for syslog logs.
For more information, see Collector Logging.
Resource mapping errors
When resource mapping fails, the Collector will ignore the syslog messages from these devices and will not forward the logs to the ingestion endpoint until a configurable time interval has elapsed.
This time interval can be configured in
agent.conf and defaults to 60 mins:
If this doesn’t help, you can check for the following messages in the Collector’s
- Added the rejected resource ID in Collector cache
- Host id is there in the rejected list since device mapping failed. Ignoring the messages
wrapper.log may also return the following response codes from the log ingestion API:
400 Bad Request
DNS resolution failure
DNS name resolution for the hostname received in the syslog message header will be performed by the Collector.
If the Collector fails to retrieve the correct hostname:
- Check the values of the following
wrapper.logfor the following message: Host is not present so ignoring this message
Log ingestion issues
If device mapping and hostname retrieval are correct, you may want to check that the logs are being ingested.
To do this, search the Collector logs for the Info message:
Response received from Ingest API. This message will include the response code and request ID from the ingestion endpoint.
For more information, see Log Ingestion API Response Codes.
Custom date and time formats
If the date and time format of the logs is not supported, LogicMonitor may not be able to parse the logs properly. Custom date and time formats can be specified in the Collector’s
If you have logs with the following date format (such as from Palo Alto Networks Panorama devices):
[2020-09-18 10:03:58.434 UTC]
You may need to add a custom date format: