Come join our live training webinar every other Wednesday at 11am PST and hear LogicMonitor experts explain best practices and answer common questions. We understand these are uncertain times, and we are here to help!
Syslog (syslog, rsyslog, syslog-ng) is one of the most common sources of log data in enterprise environments. The LogicMonitor Collector has the capability to receive Syslog data and forward the raw logs to the LM Logs Ingestion API.
Note: The Collector can be used for alerting on Syslog metrics or act as a Syslog Log Collector for LM Logs. It cannot do both.
Warning: If you are already using the LogicMonitor Collector, this is different from the Syslog EventSource configuration for alerting on Syslog metrics.
As with previous versions, the Collector can be configured to receive syslog messages by listening on port 514/UDP:
To turn on Syslog forwarding for LM Logs, uncomment the following property in the Collector’s agent.conf. If the property doesn’t exist, you can add it:
agent.conf
# Enable LM Logs lmlogs.syslog.enabled=true
Once Syslog events are received, the Collector will map them to existing monitored resources before forwarding them on to the logs ingestion API. The Collector will not forward syslog messages that cannot be associated with existing monitored resources.
For the syslog logs sent from the Collector, uncomment the following properties in agent.conf for device mapping:
system.displayname
lmlogs.logsource.syslog.unmapped.resource=system.displayname:collector_syslog
If the value of system.hostname for your monitored resources does not match their IPs reported in the Syslog events, then these default settings will not work and device mapping will fail. In this case, you will need to change the lmlogs.syslog.property.name to a property which matches the value of the lmlogs.syslog.hostname.format setting.
system.hostname
lmlogs.syslog.property.name
lmlogs.syslog.hostname.format
If you do not have an existing LogicMonitor property matching any of these values, you can create a PropertySource to add the attribute across your environment or set the property manually.
Note: If you are forwarding syslog logs from Cisco devices, the LogicMonitor Collector will attempt to use the socket address if the IP or hostname are not available. For this to work, the Collector receiving the logs needs to be monitoring the Cisco devices sending the logs.
We recommend that you configure filters to remove log messages that contain sensitive information (such as credit cards, phone numbers, or personal identifiers) so that they are not sent to LogicMonitor. Filters can also be used to reduce the volume of non-essential syslog log messages that are sent to the logs ingestion API queue.
The filtering criteria are based on the syslog fields: severity, message, facility, and application. To configure a filter criteria, uncomment to add or edit the following entries in agent.conf:
severity
facility
message
application
If the date and time format in the logs is not supported or the time information is missing, LogicMonitor may not be able to parse the logs properly. When this happens you can either specify a custom date and time format or use the timestamp of the event when it’s received. These settings can be specified in the Collector’s agent.conf.
For example, if you have logs with the following date format (such as from Palo Alto Networks Panorama devices): [2020-09-18 10:03:58.434 UTC]; you may need to add the custom date format below:
[2020-09-18 10:03:58.434 UTC]
eventcollector.syslog.supportDateFormat.8=yyyy-MM-dd'T'HH:mm:SSxxx
If you have syslog events that do not include time information, enable the following setting to use the timestamp when the log was received by the Collector:
lmlogs.syslog.useTimestampWhenCollectorReceivedLogs=TRUE
If you’ve enabled syslog log forwarding on the Collector, but the logs are not displaying in the Logs Page, you can perform the following checks to troubleshoot the issue.
Note: Collector configuration changes are universal on the Collector and can affect every device sending logs to that Collector. Changing a flag may fix one device but could affect others. It is usually a combination of changes that need to be done to resolve all syslog device issues.
In the Collector’s configuration file (agent.conf):
1. Is LM Logs enabled?
Check that lmlogs.syslog.enabled=true and that the values for lmlogs.syslog.property.name and lmlogs.syslog.hostname.format match.
lmlogs.syslog.enabled=true
Note: lmlogs.syslog.property.name can be set to ANY property in LogicMonitor. The default is system.hostname, but system.sysname or displayname can also help if naming and device configurations are inconsistent.
2. Are any filters defined that may be responsible for filtering out the logs you expect to see?
3. Were there mapping errors?
Check the Collector events to identify mapping errors.
4. Are the Logs in a timezone >= 3 hours from the Collector?
LM Logs will not accept logs that are more than 3 hours from the current. To avoid this issue, you can set the timestamp to the time the Collector receives the log events with lmlogs.syslog.useTimestampWhenCollectorReceivedLogs=true.
5. Check the Collector version and sizing.
Refer to the Collector Capacity for syslog deployments. If needed, implement a standalone LM Logs Collector which is not managing any other resources.
6. Check Collector Events.
In Manage Collector, expand the Support dropdown and select Collector Events. You can review mapping errors and search using keywords.
7. Check the Collector LM Logs graphs and Metrics.
Expand the Collector Resource > Collector DataSource Group > Collector LM Logs DataSource.
1. Adjust the logging level on the Collector to debug and review wrapper.log.
Note: Don’t forget to change the logging level back to info when you’re done troubleshooting.
2. Live tail wrapper logs.
!tail ..\logs\wrapper.log 500 syslog
In This Article