Filtering Kubernetes Resources for Monitoring

Last updated on 20 January, 2023

Argus discovers all Kubernetes resources that are monitored by LogicMonitor. In addition, you can use Argus to exclude resources from monitoring.

You can add resource filters in the Argus configuration files under the filter parameters. The rule engine evaluates rules sequentially; when the first rule is met and is evaluated as true, the rule engine breaks the rules evaluation, and the resource is excluded.

Following is the sample configuration:

FiltersDescription
'type in ("pod", "deployment") && name =~ "nginx"'If the name of the pods and deployments contains nginx, those pods and deployments will get excluded.
'!(type == "pod" && namespace == "kube-system")'Negates rule to simplify as whitelist rule; only pods of kube-system namespace will get monitored, remaining pods gets excluded.
'namespace == "default"'
Excludes all resources of the default namespace.
'!(type != "replicaset" && ns == "logicmonitor")'Excludes replicasets within LogicMonitor namespace – remaining resources will be included. In addition, this rule is equivalent to 'type == "replicaset" && ns == "logicmonitor"

Note: If you want to exclude all the resources of any type, then instead of adding a rule with a wildcard notation “name =~.*”, you must add the resources in the monitoring disable list:

Example:
argus:
monitoring:
disable:
– nodes
– services
– deployments

LogicMonitor uses the open-source Govaluate library for evaluating the filter expressions.  For more information on the syntax of the filter expressions, see the Govaluate expressions manual.

Rule Engine Variables

You can write filter rules using the following variables that are made available to the rule engine:

Variable NameValueValue DatatypeComments
typeResource TypestringThe following operators will work on the type variable: "==", "!=", "in"
Note: (known issues): in operator on the type variable doesn’t work when the array has only one element.
nameResource NameString
namespaceResource NamespaceStringEmpty, if the resource is not namespace scoped.
Resource labels with their keys as variable namesResource Labels values against its keysStringNote: As per the Govaluate documentation, you need to escape variable names having special characters viz. dot (.), hyphen (-), etc. using square brackets around it.
Resource annotations with their keys as variable namesResource Annotations values against its keysStringNote: As per the Govaluate documentation, you need to escape variable names having special characters viz. dot (.), hyphen (-), etc. using square brackets around it.

Note: If the same key is used for annotations and labels, the label value gets higher priority and is used for evaluation.

Rule Writing Guidelines

  1. Rules must be written in single-quoted strings to avoid parsing errors across YAML conversions.
  2. There must be no distinction such as include rules and exclude rules. If the rule is evaluated as true that means resources will get excluded.

Note: In some cases, if you want to simplify the rule, you can write the include rule and invert its value to make it an exclude rule.

Example 1:

If you want to monitor only resources of a single namespace frontend, the rule is

'!(ns == "frontend")'

Note: As per the Govaluate documentation, you need to escape variable names having special characters viz. dot (.), hyphen (-), etc. using square brackets around it.

Example 2

You have added a web service label on resources having their respective value. If you want to monitor only the user web service resources while excluding the remaining resources of all remaining services, then you can write the rule as '!([web-service] == "user")' – here square brackets define everything within it as a variable name while parsing the rule. If you miss surrounding the web-service variable then Govaluate makes it a mathematical expression web -(minus) service which will not exclude the resources as expected.

Example 3

The following example presents a few possible configurations you can use to selectively monitor resources in your cluster:

filters:
# Remove NGINX pods and deployment from monitoring
- 'type in ("pod", "deployment") && name =~ "nginx"'
# Remove pods in kube-system namespace from monitoring
- '(type == "pod" && namespace == "kube-system")'
# Remove resources in the default namespace from monitoring
- 'namespace == "default"'
# Remove relicasets in the logicmonitor namespace from monitoring
- '(type != "replicaset" && ns == "logicmonitor")'

Available Operators to Write Rules

OperatorDescriptionCommentsExamples
==EqualityExact string matchns == "default"
!=InequalityIs not equal to the exact stringname != "nginx"
=~Regex matchRegex having a dot and hyphen may not work in some casesname =~ "nginx" Resources having prefixes as Nginx in their name, then the resources will get excluded
!~Inverted regex patternEquivalent to !(<regex>)name !~ “nginx” equivalent to !(name =~ “nginx”)Resources that do not have Nginx in the name will be excluded
&&Logical ANDShort circuits if the left side expression is falsens == "default" && name =~ "nginx". This will exclude resources of the default namespace that has Nginx in the name
||Logical ORShort circuits if the left side evaluates to true. Although the operator is available, you must write another rule if the left side and right side are not logically connected, as the set of rules are OR’ed.
inMembership in arrayPerforms equality == to check membershipns in ("kube-system", "kube-public")This will exclude the resources of mentioned namespaces
()Parenthesis to group
In This Article