Argus discovers all Kubernetes resources that are monitored by LogicMonitor. In addition, you can use Argus to exclude resources from monitoring.

You can add resource filters in the Argus configuration files under the filter parameters. The rule engine evaluates rules sequentially; when the first rule is met and is evaluated as true, the rule engine breaks the rules evaluation, and the resource is excluded.

Following is the sample configuration:

If you want to exclude all the resources of any type, then instead of adding a rule with a wildcard notation “name =~.*”, you must add the resources in the monitoring disable list:

Example:

argus:
    monitoring:
       disable:
            - "replicasets"

LogicMonitor uses the open-source Govaluate library for evaluating the filter expressions.  For more information on the syntax of the filter expressions, see the Govaluate expressions manual.

Rule Engine Variables

You can write filter rules using the following variables that are made available to the rule engine:

Rule Writing Guidelines

1. Rules must be written in single-quoted strings to avoid parsing errors across YAML conversions.
   
2. There must be no distinction such as include rules and exclude rules. If the rule is evaluated as true that means resources will get excluded.

Example 1:

If you want to monitor only resources of a single namespace frontend, the rule is

'!(ns == "frontend")'

Note: As per the Govaluate documentation, you need to escape variable names having special characters viz. dot (.), hyphen (-), etc. using square brackets around it.

Example 2: 

You have added a web service label on resources having their respective value. If you want to monitor only the user web service resources while excluding the remaining resources of all remaining services, then you can write the rule as '!([web-service] == "user")' – here square brackets define everything within it as a variable name while parsing the rule. If you miss surrounding the web-service variable then Govaluate makes it a mathematical expression web -(minus) service – which will not exclude the resources as expected.

Example 3

The following example presents a few possible configurations you can use to selectively monitor resources in your cluster:

filters:
# Remove NGINX pods and deployment from monitoring
- 'type in ("pod", "deployment") && name =~ "nginx"'
# Remove pods in kube-system namespace from monitoring
- '(type == "pod" && namespace == "kube-system")'
# Remove resources in the default namespace from monitoring
- 'namespace == "default"'
# Remove relicasets in the logicmonitor namespace from monitoring
- '(type != "replicaset" && ns == "logicmonitor")'

Available Operators to Write Rules

Argus is a tool for monitoring Kubernetes with LogicMonitor. Argus runs as a Pod in your cluster and uses LogicMonitor’s API to add Nodes, Pods, Services, and resources into monitoring. Once in monitoring, data collection starts automatically. Argus uses Kubernetes API to collect data for all monitored resources. Additionally, you can leverage LogicMonitor DataSources to monitor your applications running within the cluster.

Argus offers the following key features for Kubernetes monitoring:

• Automated Resource Discovery: Argus leverages Kubernetes events and LogicMonitor’s API to provide real-time accuracy of the resources of a cluster in LogicMonitor. With Argus, you can discover etcd members, cluster Nodes, Services, Pods, and resources. You can also automate the management of their lifecycle as Resources in LogicMonitor.
  
• Comprehensive Monitoring: Argus manages the Dockerized LogicMonitor Collectors running in a StatefulSet and collects data via the Kubernetes API for Nodes, Pods, Services, Containers, and resources. 
  

You can install Argus either by using the command-line interface (CLI) or from your LogicMonitor portal.

Requirements

To install Argus, you need the following:

• Helm 3
  
• Kubernetes v1.14 or later
  
• Add LogicMonitor chart repository:
  
  $ helm repo add logicmonitor https://logicmonitor.github.io/k8s-helm-charts
  
• Ensure to install the LogicMonitor Collectorset-controller before installing Argus. For more information, see Installing Collectorset-controller.
  
• Refer to the Configuration templates of Helm charts values.yaml. to select compatible versions of the Collectorset-controller and Argus binaries.

Installing Collectorset-Controller

A Collectorset-Controller automatically installs and manages the LogicMonitor’s Collectors based on the requirements added in the configurations file.

Requirements:

• Helm chart (refer to Configuration templates of Helm charts values.yaml)
• LogicMonitor account
  

1. Navigate to Resources > Add and select “Kubernetes Cluster”.

2. On the Add Kubernetes Cluster page, enter the required details and click Next.

3. In the Install Argus section, select “Edit Configurations” to enter the required configuration details.

4. Update the configuration parameters in the configuration file.

5. Export the configuration file path and enter the following helm command:

$ export COLLECTORSET_CONTROLLER_CONF_FILE=<collectorset-controller-configuration-file-path>
$ helm upgrade \
  --install \
  --debug \
  --wait \
  --namespace="$NAMESPACE" \
  -f "$COLLECTORSET_CONTROLLER_CONF_FILE" \
  collectorset-controller logicmonitor/collectorset-controller

For more information on the list of values and their descriptions used in the collectorset-controller configuration yaml file, see Default value from ArtifactHUB.

Installing Argus

The setup wizard provides the configuration and installation commands for the applications needed for monitoring CollectorSet-Controller and Argus.

1. From the setup wizard, select “Edit Configuration” to customize the YAML configuration files for the CollectorSet-Controller and Argus. For the complete list of configuration options, see Configurations.

2. Select “Install” to see the Helm commands to install the CollectorSet-Controller and Argus and LogicMonitor’s Helm Charts. You can copy and paste the commands to install the integration into your cluster.

For more information on the list of values and their descriptions used in the argus configuration yaml file, see Default value from ArtifactHUB.

3. Click Verify Connection to ensure that the Collectors and the cluster resources were properly added into monitoring.

Configuring Collectorset-Controller using Helm Charts

The Collectorset-Controller Helm chart supports the following values:
Required Values

Optional Values

Parameters Settings

Advanced Parameters Settings

Configuring Argus using Helm Chart

The Argus Helm chart supports the following values:

Required Values

Optional Values

Advanced Parameter Settings

With Argus v5 or the previous resource tree, dynamic groups were created for each type of resource. Now with Argus v7, the resource tree is enabled by default, to optimize the performance. In the new resource tree, all the namespace scoped resources such as Pods, Deployments, and Services are grouped under Namespaces, and cluster scoped resources such as etcd is grouped under the ClusterScoped group.

Updating Resource Tree

To enable the new resource tree structure, set enableNewResourceTree: true in the argus-config.yaml file. Following are the samples of Argus v5 and v6 resource tree structure :Argus v5 resource tree sample:

Argus v6 resource tree sample:

Once you have successfully made the changes to the resources in the new structure, you may delete the old resource tree structure.

When you add your Kubernetes cluster into monitoring, dynamic groups are used to group the cluster resources. For more information on adding a Kubernetes cluster into monitoring, see Adding Kubernetes Cluster into Monitoring.

Earlier, for adding dynamic groups, you required ‘manage all resources’ permission. However, now you can use API keys that have access to at least one resource group to add clusters for monitoring. The dynamic groups are linked to the resource groups that have the API keys with view permissions.

Administrator: Steps to Assign Permission to a Non-Admin User

1. Create different device groups for non-admin users. For more information, see Adding Device Groups.

2. Navigate to Settings> User Access> User and Roles> Roles tab.
   
3. Select the required user group and click the Gear icon.
   
4. On the Manage Role dialog box, provide access to the respective static groups.
   Note: You can create multiple users with specific roles from the Manage Role dialog box.
   
   Once the required permissions are provided, the non-admin users can add and monitor the Kubernetes clusters within the static groups respectively.
   
   
   
5. Create different Collector groups and Dashboard groups for the non-admin users and provide access to the users for their respective groups.
   
6. Select the User Profile setting and grant access to the non-admin users to create the API tokens and manage their profiles.
   
   

Non-admin user: Adding a Kubernetes Cluster

Once the administrator has completed all the prerequisites and allocated a resource group, then you can complete the following steps to add the Kubernetes cluster:

1. Navigate to Resources > Devices > select the allocated resource group for adding the cluster.

2. From the Add drop-down list, select Kubernetes Cluster.

3. On the Add Kubernetes Cluster page, add the following information:

a. In the Cluster Name field, enter the cluster name.

b.In the API token field, select the API token for the allocated resource group and click Save.
The other API Token fields information will get populated automatically.

c. In the Resource Group field, select the allocated resource group name.

d. In the Collector Group and Dashboard Group fields, select the allocated Resource Group.

4. Click Next.

5. In the Install Instruction section, select the Argus tab.

6. Select the resourceGroupID parameter and replace the default value with the system.deviceGroupId property value of the allocated resource group.

7. Click Verify Connection. Once the connection is successful, your cluster is added.

Google Anthos enables you to run and manage Kubernetes Clusters running anywhere, whether natively via Google Kubernetes Engine (GKE) or externally in another cloud service and on-premise.

With LM Container, you can ensure central and consistent monitoring for clusters managed through Anthos. Specifically, LM Container provides a central view within LogicMonitor for you to monitor the health and performance of your cluster components (such as nodes, pods, containers, services, deployments, and so on) as well as the performance of the applications running within the clusters.

To get started, add each of your clusters into monitoring.

Once added, you should see the Anthos-managed clusters within LogicMonitor. Container applications will be automatically detected and monitored.

Alternatively, you can develop your own LogicModules to monitor custom applications.

To get additional metrics for Google Cloud Platform (GCP) resources, such as GKE notes, you can add your GCP projects into monitoring.

LogicMonitor’s Kubernetes monitoring involves two semantically versioned applications: Argus and the Collectorset-Controller. New versions of these applications provide bug fixes, improvements, and new features. You can see the latest available versions of these applications (and the enhancements they provide) on their respective GitHub host pages.

• Argus
• Collectorset-Controller

Release Tagging

Argus images are tagged with semantic versioning in addition to a tag for the major version (for example, v5) which will match all images that get published with that major version (including minor improvement and bug fix increments). This ensures that new images with minor improvements and bug fixes are used automatically.

For example, for Argus version 5.0.0, the following tags are applied to the image: “v5” and “5.0.0”. If you reference “v5” in your Helm deployment at the time 5.0.0 is released, that tag would also match subsequent Argus versions 5.1.0, 5.1.1, and so on upon their release.

Opt-in is still required when a new major version is released. This means that if your Helm deployment for Argus and Collectorset-Controller currently references “latest” or hard-coded version image tags, you will need to opt into the upgrade using the instructions provided in the next section.

Opting Into Major Version Changes

Prerequisites

Ensure the following requirements are met before upgrading:

• Helm 3
• Kubernetes 1.14+

Upgrade Instructions

To upgrade the Argus chart from older versions to the latest version, you require an Argus configuration file for running the helm upgrade command.

Creating Argus configuration file

For creating .yaml configuration files for argus and collector-set controller, complete the following steps:

1. Download the configuration file template from here.

2. Get existing values using the command:

helm get values argus 

3. Save in the backup configuration file.

4. Put all existing values in the downloaded configuration file at their appropriate places. And leave the remaining values at their default.

Upgrading the Argus using Helm deployment

For upgrading to the latest version of Argus, complete the following steps:

1. Run helm repo update.
   
2. To upgrade the Argus helm chart, run the following command:
   
   helm upgrade -f argus-config.yaml argus logicmonitor/argus


Updating Configuration Parameters in the Configuration File

Export the configuration file path and enter the following helm command:

$ export ARGUS_CONF_FILE=<argus-configuration-file-path>
$ helm upgrade \
  --install \
  --debug \
  --wait \
  --namespace="$NAMESPACE" \
  -f "$ARGUS_CONF_FILE" \
  argus logicmonitor/argus

See the Configurations section for a list of values the Argus helm chart supports and their descriptions.

Configuration templates of Helm charts values.yaml

Collectorset Controller

Latest Collectorset-controller-config.yaml

Argus

Latest argus-config.yaml

Overview

​Istio is a service mesh that provides traffic management, policy enforcement, and telemetry collection for microservices. Using LogicMonitor’s Istio package for Kubernetes, you can gather metrics from the backing Prometheus time-series database (TSDB) that comes bundled with Istio. LogicMonitor’s Istio Kubernetes package supports Kubernetes only.

​The LogicModules in the Istio Kubernetes package utilize the “/metrics” endpoint from the Prometheus pod on port 9090. By default, these LogicModules apply to the pod in the “istio-system” namespace labeled with “app=prometheus”.

Compatibility

LogicMonitor’s Istio Kubernetes package supports Kubernetes only. As Istio releases support for other platforms, LogicMonitor will test and extend coverage as necessary.

Setup Requirements

Import LogicModules

From the LogicMonitor Repository, import all Istio Kubernetes LogicModules, which are listed in the LogicModules in Package section of this support article.

Add Devices Into Monitoring

It is recommended that your Kubernetes cluster already be added into LogicMonitor for monitoring, as many of the AppliesTo properties set by Argus, LogicMonitor’s open-source Kubernetes monitoring solution, are necessary for seamless operation. For instructions on adding your Kubernetes cluster into monitoring, see Adding your Kubernetes Cluster into Monitoring.

Port Access

The Istio Kubernetes package gathers metrics from the backing Prometheus TSDB that comes bundled with Istio. Istio must be installed on the Kubernetes cluster and port 9090, which is used to access the Prometheus “/metrics” endpoint, must be open to the Collector.​

LogicModules in Package

​LogicMonitor’s Istio Kubernetes package consists of the following LogicModules. For full coverage, please ensure that all of these LogicModules are imported into your LogicMonitor platform.

When you initially add your cluster into monitoring via the ‘Add Kubernetes Cluster’ wizard, you configure various aspects of monitoring (Collector size, whether Kubernetes RBAC is enabled, how many Collector replicas you want, etc.) and LogicMonitor provides you with Helm commands that enable install matching your configuration:

This wizard is not available after adding your cluster into monitoring. If you need to change one of these configuration items later, the most straightforward way to do so is via Helm directly in your cluster. Specifically, you’ll need to upgrade your Helm release for Argus and/or the Collectorset-Controller and change the desired options.

Note: If Collector pods attempt to recreate with a Collector version that is no longer available, the newest Collector version will be used automatically to ensure successful pod start-up.

Configuration Options

You can see all available configuration options for Argus here, and for the Collectorset-Controller here.

We recommend using the --reuse-values flag during the upgrade to ensure previous options that aren’t changed remain as-is. For example, to update the number of Collector replicas to ‘2’, you might run the following in your cluster:

helm upgrade --reuse-values --recreate-pods --set collector.replicas="2" argus logicmonitor/argus

You can also use Helm to view the current configuration for the Argus and Collectorset-Controller releases, which may be helpful before making changes:

helm get values argus

Note: LogicMonitor updates the Helm Charts repository for certain releases. To ensure you have the latest charts in your cluster, run the following command before making changes:

helm repo update

Deleting LM Container

You may want to start with a fresh install. To remove existing Argus and Collectorset-Controller from a Kubernetes cluster, run the following Helm commands:

helm uninstall collectorset-controller
helm uninstall argus

Kubernetes Resource Visibility Limit

Addition of excessive Kubernetes resources causes service disruptions and degrade portal performance. To support platform performance and optimize the user experience, LogicMonitor enforces a limit on the number of Kubernetes resources.

LogicMonitor enforces this limit starting with v.225. 

Requirements for Resolving Kubernetes Resource Visibility Limit

If the platform-wide limit of 25,000 resources is reached, additional Kubernetes resources are excluded by the system and do not display in the platform.

Before troubleshooting, review the following requirements:

• The default platform-wide limit for Kubernetes Resources is 25,000.
• Additional Kubernetes resources beyond this limit are not displayed in the platform.
• The limit applies to all resources, including those that have been deleted but are retained based on retention policy

Resolving the Kubernetes Resource Visibility Limit

If the resource limit is reached and additional Kubernetes resources do not display, do the following:

1. Review the Resource Retention Policy
   Check the globalDeleteAfterDuration property in the lm-container-configuration.yaml file to see your current retention configuration or review the setting in the LogicMonitor portal. You can also check the kubernetes.resourcedeleteafterduration property in the Kubernetes root folder.

2. Adjust the Retention Period
   If the currently set retention limit is too long, adjust it to a more suitable value. Shortening the retention period reduces the number of retained (deleted) resources counted toward the limit.
   
3. Contact Customer Support Manager
   If your operational needs require a higher threshold than 25,000 resources, contact your Customer Success Manager to discuss available options.

If the Helm install commands fail:

• Make sure that the Kubernetes RBAC enabled option was correctly selected during install. This controls the ‘enableRBAC’ Helm chart values, which must match the RBAC in your cluster (for example if RBAC is not enabled for your cluster, this value must be false).
• Confirm that your LogicMonitor API Tokens are valid, and have sufficient permission.
• Check the logs for the Argus & Collectorset-Controller pods

If Helm install commands succeed, but your cluster is not added to monitoring:

• Check whether the Collector Group added. If it wasn’t, there may be an issue with the LogicMonitor API Tokens provided. You can use the LogicMonitor Audit Log to identify if the request was denied.
• Check whether Collectors were successfully installed. If the Collector Group was added, but Collectors weren’t successfully installed, it’s not likely to be an issue with your API Tokens. It may be that Collector installation memory requirements were not met – confirm that this is not the case. If the Collectors aren’t properly installed, it’s also a good idea to look at the logs for the Argus, Collectorset-Controller, and Collector pods.
• If you are using OpenShift, you may need to elevate the permissions of the serviceaccount for the Collector to enable install. You can do this via the following command (assumes the default namespace is used):
  oc adm policy add-scc-to-user anyuid system:serviceaccount:default:collector

If Helm install commands succeed, and your cluster was added into monitoring but data collection does not work:

• Ensure you’re using the latest version of the DataSources. We regularly release new versions of our DataSources to include bug fixes and improvements. To import newer versions, please see this article
• If, with the latest version of the DataSources, data collection still doesn’t work – you can modify the DataSource to display the Kubernetes API response and status code to get more information. For example, the Pod DataSource could be modified to print out request and response information to narrow down what error the Kubernetes API is returning during data collection attempts, as in the following screenshot. This may help you identify why data collection isn’t working. 

If you are unable to set up Argus and Collectorset-Controller pods in GKE due to memory, CPU, and ephemeral storage, use the following configuration:

• Ensure to set the resource limits and requests (memory, CPU, ephemeral storage) for the collector using the Argus configuration file.
  Sample file:

statefulsetspec:
      template:
        spec:                                                                                                                                                                               
         containers:
           - name: collector
           resources:
             requests:
               cpu: 1000m
               ephemeral-storage: 5Gi
               memory: 2Gi

If the following gRPC connection failure occurs while installing Argus:

level=warning msg="Error while creating gRPC connection. Error: context deadline exceeded" argus_pod_id=<pod-id> debug_id=<id> goroutine=1 method=pkg/connection.createGRPCConnection watch=init

1. Run the following command to log in to the Argus Pod shell:
   
   kubectl exec -it <argus_Pod_name> /bin/sh

2. Check communication between the Argus and Collectorset-Controller pods:
   
   curl http://collectorset-controller:50000

3. If the communication fails and an error occurs, you must check the parameters of the restrictions set in the internal network policies.

If Collector pods restart frequently on an OpenShift v4.x cluster and cause monitoring gaps, do the following:

The Docker collector runs all the collection jobs; however, the limit is insufficient for the large-scale clusters as OpenShift has a default container PID limit of 1024 that limits the number of processes to 1024.

You can modify the settings by using a ContainerRuntimeConfig custom resource.

For example, you have labeled a Pod as machineconfigpool on which you want to increase the PID limit with key as custom-crio and value as custom-pidslimit, then modify the configuration file as follows:

apiVersion: machineconfiguration.openshift.io/v1
kind: ContainerRuntimeConfig
metadata:
 name: custom-pidslimit
spec:
 machineConfigPoolSelector:
   matchLabels:
     custom-crio: custom-pidslimit
 containerRuntimeConfig:
   pidsLimit: 4096

You can also verify if the PID limit is effective by entering the following command in the pod shell:
cat /sys/fs/cgroup/pids/pids.current