Support Center Home


Collecting and Forwarding Kubernetes Events

You can configure the LogicMonitor Collector to receive and forward Kubernetes Events and Pod logs to the LM Logs ingestion API.

Prerequisites

Add Resources to the Collector for monitoring

Note: This section only applies to existing clusters in monitoring. You do not need to make this edit if the cluster was just added into monitoring with the lastest version of Argus.

The Cluster Role Collector needs to have access to the resources you want to monitor.

$ kubectl edit clusterrole collector

Under apiGroups > resources, add events and pod/logs. For example:

- apiGroups:
  resources:
  - events
  - pod/logs

Enable logs collection

You have two options for enabling logs collection.

1. (Recommended) Modify the Helm deployment for Argus to enable events collection.

helm upgrade --reuse-values \ 
   --set device_group_props.cluster.name="lmlogs.k8sevent.enable" \ 
   --set device_group_props.cluster.value="true" \ 
   --set device_group_props.pods.name="lmlogs.k8spodlog.enable" \ 
   --set device_group_props.pods.value="true" \ 
argus logicmonitor/argus

2. Manually add the following properties to the monitored Kubernetes cluster group (or individual resources) in LogicMonitor.

Property Description
lmlogs.k8sevent.enable=true Sends events from pods, deployments, services, nodes, and so on to LM Logs. When false, ignores events.
lmlogs.k8spodlog.enable=true Sends pod logs to LM Logs. When false, ignores logs from pods.

Optional configurations

You can add or edit the following entries in the Collector’s agent.conf:

Property Description Default
lmlogs.k8sevent.polling.interval.min=1 Polling interval in minutes for Kubernetes events collection. 1
lmlogs.k8spodlog.polling.interval.min=1 Polling interval in minutes for Kubernetes pod logs collection. 1
lmlogs.thread.count.for.k8s.pod.log.collection=20 Number of threads for Kubernetes pod logs collection. The maximum value is 50. 10

Configure filters to remove logs

We recommend that you configure filters to remove log messages that contain sensitive information (such as credit cards, phone numbers, or personal identifiers) so that they are not sent to LogicMonitor. Filters can also be used to reduce the volume of non-essential syslog log messages that are sent to the logs ingestion API queue.

The filtering criteria for Kubernetes Events are based on the fields: message, reason, and type. For Kubernetes pod logs, you can filter on the message fields. Filtering criteria can be defined using keywords, a regular expression pattern, specific values of fields, and so on. To configure a filter criteria, uncomment to enable and then edit the filtering entries in agent.conf. For example:

  • To filter out INFO level pod logs to LogicMonitor, uncomment or add the line: logsource.k8spodlog.filter.1.message.notcontain=INFO
  • To send Kubernetes events of type=Normal, comment out the line: logsource.k8sevent.filter.1.type.notequal=Normal

Troubleshooting

If you are not receiving pod logs, restart the Collector and increase the polling interval to 3-5 minutes.

In This Article