Sending Syslog Logs
Last updated on 25 August, 2024Syslog (syslog, rsyslog, syslog-ng) is one of the most common sources of log data in enterprise environments. Syslog data originates in the infrastructure and is forwarded to a target IP address which for LogicMonitor is the Collector. The Collector currently monitoring the resource does not have to be same one that receives the Syslog messages.
Recommendation: LogSource is the recommended method to enable LM Logs. To use LogSource, the LM Collector must be version EA 31.200 or later. For more information, see LogSource Overview, or contact your Customer Success Manager. The procedure in the following describes how to enable LM Logs if you are not using LogSource.
Note: An LM Collector can be used for alerting on Syslog metrics using EventSources, or act as a Syslog log collector for LM Logs ingestion and processing. It cannot do both.
Enabling LM Logs Syslog collection on a Collector disables all Syslog-based EventSources on that Collector. However, since the polling Collector and the Syslog Collector don’t have to be the same, you can set up a dedicated Syslog Collector for LM Logs collection, and leave the Syslog-based EventSource monitoring enabled on the specific Collector monitoring that device.
Requirements
- EA Collector 29.104 or later installed.
- Syslog collection and log forwarding requires port 514/UDP to be open and available on both the Collector machine and firewall. Note: This may require disabling UDP for rsyslog.
Enabling Syslog Log Forwarding
The LM Collector receives Syslog data and forwards the raw logs to the LM Logs Ingestion API for viewing, searching and analysis.
As with previous versions, the Collector can be configured to receive syslog messages by listening on port 514/UDP:
- Devices monitored by the Collector can forward syslog messages from an existing centralized syslog server.
- Monitored devices that have a syslog client can be configured to send the syslog data to the Collector.
- Monitored devices are not required to send syslog messages to the same Collector that they are monitored by.
To turn on Syslog forwarding for LM Logs, uncomment the following property in the Collector’s agent.conf
. If the property doesn’t exist, you can add it:
# Enable LM Logs
lmlogs.syslog.enabled=true
Configuring Resource Mapping
Once Syslog events are received, the Collector will map them to existing monitored resources before forwarding them on to the logs ingestion API. If the log cannot be mapped to an LM-monitored resource, you can still view and and search the log by “_resource.id=0”.
For the syslog logs sent from the Collector, uncomment the following properties in agent.conf
for device mapping:
Property | Description | Default |
lmlogs.syslog.property.name | Resource properties which we want to use for device mapping. | system.hostname |
lmlogs.syslog.hostname.format | The expected format of the resource property specified in lmlogs.syslog.property.name. Possible values are IP, FQDN, HOSTNAME, DO_NOTHING. If set to IP, HOSTNAME, or FQDN, the syslog message will be resolved by making a DNS query. If set to DO_NOTHING, then DNS resolution does not happen and the received hostname is set to the resource ID. |
IP |
lmlogs.syslog.UseHostNameFromMessages | Property to use the hostname from messages. | false |
lmlogs.syslog.UseSocketAddressIfhostNameisInvalid | If the hostname is not valid, you may choose to use the socket address for device mapping. | true |
lmlogs.logsource.syslog.unmapped.resource | If resource mapping fails, you may choose a backup resource to associate logs to. One backup resource can be defined for each Collector, and all logs that fail to map to its resource for any reason, will be mapped to this backup resource. When configuring a global backup resource, you can choose any resource property such as system.displayname followed by a colon and the value of the LogicMonitor property. For example, to configure “collector_syslog” as the backup resource:lmlogs.logsource.syslog.unmapped.resource=system.displayname:collector_syslog
|
N/A |
If the value of system.hostname
for your monitored resources does not match their IPs reported in the Syslog events, then these default settings will not work and device mapping will fail. In this case, you will need to change the lmlogs.syslog.property.name
to a property which matches the value of the lmlogs.syslog.hostname.format
setting.
If you do not have an existing LogicMonitor property matching any of these values, you can create a PropertySource to add the attribute across your environment or set the property manually.
Note: If you are forwarding syslog logs from Cisco devices, the LM Collector will attempt to use the socket address if the IP or hostname are not available. For this to work, the Collector receiving the logs needs to be monitoring the Cisco devices sending the logs
Configuring Filters to Remove Logs
We recommend that you configure filters to remove log messages that contain sensitive information (such as credit cards, phone numbers, or personal identifiers) so that they are not sent to LogicMonitor. Filters can also be used to reduce the volume of non-essential syslog log messages that are sent to the logs ingestion API queue.
The filtering criteria are based on the syslog fields: severity, message, facility, and application.
Note: The following filters are not supported for the collector version EA-35.200 and above. You can add the filters to the LogSource to maintain the same level of filtering. For more information, see Syslog LogSource Configuration.
To configure a filter criteria, uncomment to add or edit the following entries in agent.conf
:
Filter Name | Description | Examples |
severity |
Define filters for syslog messages with a severity level equal to or more urgent than the specified value. Severity levels can be (in descending order of urgency): Emergency, Critical, Alert, Error, Warning, Notify, Informational, Debug. | logsource.syslog.filter.1.severity.equal=informational logsource.syslog.filter.2.severity.moreurgentthan=informational |
facility |
Define filters for syslog messages based on the facility value: kernel messages, user-level messages, clock daemon, and so on. | logsource.syslog.filter.11.facility.equal=kernel messages logsource.syslog.filter.12.facility.notequal=clock daemon logsource.syslog.filter.13.facility.contain=NTP logsource.syslog.filter.14.facility.notcontain=NTP logsource.syslog.filter.15.facility.exist=* logsource.syslog.filter.16.facility.notexist=* logsource.syslog.filter.17.facility.greaterthan=NTP logsource.syslog.filter.18.facility.greaterequal=NTP logsource.syslog.filter.19.facility.lessthan=NTP logsource.syslog.filter.20.facility.lessequal=NTP logsource.syslog.filter.21.facility.regexmatch=Target[\S\s]*-[Ww][Aa] logsource.syslog.filter.22.facility.regexnotmatch=Target[\S\s]*kernel |
message |
Define filters for syslog messages based on the contents of the message itself, using keywords or a regular expression pattern to match. | logsource.syslog.filter.3.message.equal=negotiate IPsec phase 1 logsource.syslog.filter.4.message.notequal=negotiate IPsec phase 1 logsource.syslog.filter.5.message.contain=negotiate IPsec phase 1 logsource.syslog.filter.6.message.notcontain=negotiate IPsec phase 1 logsource.syslog.filter.7.message.regexmatch=(negotiate)+\w logsource.syslog.filter.8.message.regexnotmatch=(negotiate)+\w logsource.syslog.filter.9.message.exist=* logsource.syslog.filter.10.message.notexist=* |
application |
Define filters for syslog messages based on the application, using keywords or a regular expression pattern to match. | logsource.syslog.filter.23.application.equal=snmpd logsource.syslog.filter.24.application.notequal=snmpd logsource.syslog.filter.25.application.contain=snmpd logsource.syslog.filter.26.application.notcontain=MySQL logsource.syslog.filter.27.application.regexmatch=Target[\S\s]*-[Ww][Aa] logsource.syslog.filter.28.application.regexnotmatch=Target[\S\s]*admin logsource.syslog.filter.29.application.exist=* logsource.syslog.filter.30.application.notexist=* |
Date and Timestamp Parsing
If the date and time format in the logs is not supported or the time information is missing, LogicMonitor may not be able to parse the logs properly. When this happens you can either specify a custom date and time format or use the timestamp of the event when it’s received. These settings can be specified in the Collector’s agent.conf
.
For example, if you have logs with the following date format (such as from Palo Alto Networks Panorama devices): [2020-09-18 10:03:58.434 UTC]
; you may need to add the custom date format below:
eventcollector.syslog.supportDateFormat.8=yyyy-MM-dd'T'HH:mm:SSxxx
If you have syslog events that do not include time information, enable the following setting to use the timestamp when the log was received by the Collector:
lmlogs.syslog.useTimestampWhenCollectorReceivedLogs=TRUE
Troubleshooting
If you’ve enabled syslog log forwarding on the Collector, but the logs are not displaying in the Logs page, you can perform the following checks to troubleshoot the issue.
Collector Configuration
Note: Collector configuration changes are universal on the Collector and can affect every device sending logs to that Collector. Changing a flag may fix one device but could affect others. It is usually a combination of changes that need to be done to resolve all syslog device issues.
Check the following in the Collector’s configuration file (agent.conf):
1. Is LM Logs enabled?
Check that lmlogs.syslog.enabled=true and that the values for lmlogs.syslog.property.name and lmlogs.syslog.hostname.format match.
lmlogs.syslog.property.name can be set to ANY property in LogicMonitor. The default is system.hostname, but system.sysname or displayname can also help if naming and device configurations are inconsistent.
2. Are any filters defined that may be responsible for filtering out the logs you expect to see?
3. Were there mapping errors?
Check the Collector events to identify mapping errors.
- Change lmlogs.syslog.property.name and lmlogs.syslog.hostname.format appropriately to make sure that the right information from the syslog event is mapped against the LogicMonitor property value.
- Use lmlogs.syslog.hostname.format=DO_NOTHING to bypass DNS if hostnames are already resolved. This is not listed in the configuration file by default.
- Some resource types do not pass along the hostname with syslog metadata, but the hostname is in the message field. Set lmlogs.syslog.UseHostNameFromMessages=true to use the hostname from the message.
- If the hostname is not valid, LogicMonitor defaults to using the socket address for device mapping. Check that lmlogs.syslog.UseSocketAddressIfhostNameisInvalid=true.
4. Are the Logs in a timezone >= 3 hours from the Collector?
LM Logs will not accept logs that are more than 3 hours from the current. To avoid this issue, you can set the timestamp to the time the Collector receives the log events with lmlogs.syslog.useTimestampWhenCollectorReceivedLogs=true.
5. Check the Collector version and sizing.
Refer to the Collector Capacity for syslog deployments. If needed, implement a standalone LM Logs Collector which is not managing any other resources.
6. Check Collector Events.
In Manage Collector, expand the Support dropdown and select Collector Events. You can review mapping errors and search using keywords.
7. Check the Collector LM Logs graphs and Metrics.
Expand the Collector Resource > Collector DataSource Group > Collector LM Logs DataSource.
- If queuing issues are happening and debug logs also show an increasing queue size, adjust the size for lmlogs.thread.count.for.ingest.api.communication=10. We recommend that you double the default size (which is 10) and view the results in the graphs before increasing the size further.
- Other graphs include the number of incoming events, logs dropped, mapping errors, events ignored due to filters, and more.
Debug Logs
1. Adjust the logging level on the Collector to debug and review wrapper.log.
- In Settings > Collector > Logs > Manage, set the logging level for the eventcollector.syslog component to debug. (The default is info.)
- In Manager Collector, expand the Support dropdown and select “Send logs to LogicMonitor”. Then, review the wrapper.log files with the Log tab for the given collector.
Note: Remember to change the logging level back to info when you’re done troubleshooting.
2. Live tail wrapper logs.
- In Manage Collector, expand the Support dropdown and select “Run Debug Command”.
- Enter the following to show the last 500 lines in wrapper.log:
!tail ..\logs\wrapper.log 500 syslog
You may also change syslog to an IP address or grep another keyword.
UDP Port 514 and Firewall issues for Syslogs on Windows Collector
Due to inbound firewall rules, Syslogs may not reach the Windows collector which listens on port 514. In such a case, you can troubleshoot the issue using the following steps.
1. Check if the firewall is enabled.
In PowerShell, run the following command to check if the firewall is enabled.
$firewallStatus = Get-NetFirewallProfile | Select-Object -ExpandProperty Enabled; if($firewallStatus -eq "True") { Write-Host "Windows Firewall is Enabled" } else { Write-Host "Windows Firewall is Disabled" }
- If the Windows firewall is enabled, check if the inbound firewall rules are added to allow the Syslogs from remote hosts to reach the Windows machine.
- If the inbound rules are not added, add them for the respective remote hosts.
Alternatively, to troubleshoot the issue, you can temporarily disable the Windows firewall.
Disclaimer: LogicMonitor does not recommend to disable the Windows firewall as it can make your Windows machine vulnerable to network-based security threats.
2. Check if any security software or antivirus software is running on the Windows machine.
Your Windows machine may be running security or antivirus software configured to reject traffic on port 514. In PowerShell, to check all the processes on the Windows machine, run the following command:
Get-Process | Select-Object Name
The output displays the name of the processes running on the Windows machine. If any security software or antivirus software is running, you can change their configuration to allow traffic on port 514 from the intended IPs.
Note: A .pcap (packet capturing) file displays data received only at the data link layer. As a result, the Syslog packets available in the .pcap file are not enough to ensure that a collector listens to the Syslogs on port 514. A collector can listen to the Syslogs only when the Syslog packets are allowed at the application layer.
3. Check if any security software or antivirus software is running on the Windows machine.
The Windows collector cannot access port 514 if the port is already used by another application or service. In PowerShell, run the following command to check the process ID which is accessing the Syslogs on port 514.
netstat -ano | findstr 514
If there is another application or service, stop or kill that process and restart the collector.