Filtering Kubernetes Resources for Monitoring
Last updated on 01 November, 2023Argus discovers all Kubernetes resources that are monitored by LogicMonitor. In addition, you can use Argus to exclude resources from monitoring.
You can add resource filters in the Argus configuration files under the filter parameters. The rule engine evaluates rules sequentially; when the first rule is met and is evaluated as true, the rule engine breaks the rules evaluation, and the resource is excluded.
Following is the sample configuration:
| Filters | Description |
'type in ("pod", "deployment") && name =~ "nginx"' | If the name of the pods and deployments contains nginx, those pods and deployments will get excluded. |
'!(type == "pod" && namespace == "kube-system")' | Negates rule to simplify as whitelist rule; only pods of kube-system namespace will get monitored, remaining pods gets excluded. |
'namespace == "default"' | Excludes all resources of the default namespace. |
'!(type != "replicaset" && ns == "logicmonitor")' | Excludes replicasets within LogicMonitor namespace – remaining resources will be included. In addition, this rule is equivalent to 'type == "replicaset" && ns == "logicmonitor" |
Note: If you want to exclude only helm chart secrets, the rule is
argus:
filters:
"secret" && jsonGet(object, "type") == "helm.sh/release.v1"'
If you want to exclude all the resources of any type, then instead of adding a rule with a wildcard notation “name =~.*”, you must add the resources in the monitoring disable list:
Example:
argus:
monitoring:
disable:
- "replicasets"LogicMonitor uses the open-source Govaluate library for evaluating the filter expressions. For more information on the syntax of the filter expressions, see the Govaluate expressions manual.
Rule Engine Variables
You can write filter rules using the following variables that are made available to the rule engine:
| Variable Name | Value | Value Datatype | Comments |
| type | Resource Type | string | The following operators will work on the type variable: "==", "!=", "in" Note: (known issues): in operator on the type variable doesn’t work when the array has only one element. |
| name | Resource Name | String | |
| namespace | Resource Namespace | String | Empty, if the resource is not namespace scoped. |
| Resource labels with their keys as variable names | Resource Labels values against its keys | String | Note: As per the Govaluate documentation, you need to escape variable names having special characters viz. dot (.), hyphen (-), etc. using square brackets around it. |
| Resource annotations with their keys as variable names | Resource Annotations values against its keys | String | Note: As per the Govaluate documentation, you need to escape variable names having special characters viz. dot (.), hyphen (-), etc. using square brackets around it. |
Note: If the same key is used for annotations and labels, the label value gets higher priority and is used for evaluation.
Rule Writing Guidelines
- Rules must be written in single-quoted strings to avoid parsing errors across YAML conversions.
- There must be no distinction such as include rules and exclude rules. If the rule is evaluated as true that means resources will get excluded.
Note: In some cases, if you want to simplify the rule, you can write the include rule and invert its value to make it an exclude rule.
Example 1:
If you want to monitor only resources of a single namespace frontend, the rule is
'!(ns == "frontend")'
Note: As per the Govaluate documentation, you need to escape variable names having special characters viz. dot (.), hyphen (-), etc. using square brackets around it.
Example 2:
You have added a web service label on resources having their respective value. If you want to monitor only the user web service resources while excluding the remaining resources of all remaining services, then you can write the rule as '!([web-service] == "user")' – here square brackets define everything within it as a variable name while parsing the rule. If you miss surrounding the web-service variable then Govaluate makes it a mathematical expression web -(minus) service – which will not exclude the resources as expected.
Example 3
The following example presents a few possible configurations you can use to selectively monitor resources in your cluster:
filters:
# Remove NGINX pods and deployment from monitoring
- 'type in ("pod", "deployment") && name =~ "nginx"'
# Remove pods in kube-system namespace from monitoring
- '(type == "pod" && namespace == "kube-system")'
# Remove resources in the default namespace from monitoring
- 'namespace == "default"'
# Remove relicasets in the logicmonitor namespace from monitoring
- '(type != "replicaset" && ns == "logicmonitor")'Available Operators to Write Rules
| Operator | Description | Comments | Examples |
| == | Equality | Exact string match | ns == "default" |
| != | Inequality | Is not equal to the exact string | name != "nginx" |
| =~ | Regex match | Regex having a dot and hyphen may not work in some cases | name =~ "nginx" Resources having prefixes as Nginx in their name, then the resources will get excluded |
| !~ | Inverted regex pattern | Equivalent to !(<regex>) | name !~ “nginx” equivalent to !(name =~ “nginx”)Resources that do not have Nginx in the name will be excluded |
| && | Logical AND | Short circuits if the left side expression is false | ns == "default" && name =~ "nginx". This will exclude resources of the default namespace that has Nginx in the name |
| || | Logical OR | Short circuits if the left side evaluates to true. Although the operator is available, you must write another rule if the left side and right side are not logically connected, as the set of rules are OR’ed. | |
| in | Membership in array | Performs equality == to check membership | ns in ("kube-system", "kube-public")This will exclude the resources of mentioned namespaces |
| () | Parenthesis to group |