Kubernetes has emerged as the de facto standard for container orchestration in modern software development, allowing organizations to manage and scale containerized applications easily. As a highly dynamic and distributed system, however, Kubernetes can be challenging to manage and maintain at scale. One of the most critical aspects of maintaining a stable and secure Kubernetes cluster is monitoring the object configurations and tracking the changes over a period of time.

Kubernetes object configuration refers to the desired state of Kubernetes objects, such as deployments, services, and pods, as described in Kubernetes YAML files. Ensuring that the cluster’s actual state matches its desired state is crucial for maintaining consistency and stability. The desired state in this context refers to the state in which custom enterprise requirements expect the objects to be. An example of the desired state can be a requirement to grant only the watch permissions on pods to a user or a ConfigMap setting value that must be greater than the specified threshold. Monitoring changes to the cluster’s configuration is essential for detecting and addressing issues such as configuration drift, security vulnerabilities, and performance problems.

In this article, we will explore the importance of monitoring Kubernetes object configuration, how it can be achieved, the current tools available, and explore some of LogicMonitor’s advanced Kubernetes configuration management capabilities. We will also provide examples of how monitoring Kubernetes object configuration can help identify and resolve common issues and optimize resource usage, leading to a more streamlined and efficient DevOps process.

Ensuring Kubernetes configuration consistency with monitoring

Kubernetes object configuration defines the desired state of the objects that make up the Kubernetes cluster, including pods, deployments, services, and other objects. Configuration is specified using YAML or JSON files, and Kubernetes uses this configuration to ensure that the cluster is in its desired state. However, ensuring that the configuration remains consistent with the enterprise requirements can be challenging, especially as the size of the cluster grows.

Changes to Kubernetes object configuration can occur for various reasons, such as new deployments, updates to existing objects, or changes to the underlying infrastructure. Monitoring Kubernetes object configuration can help detect changes that were not intended, such as unauthorized modifications or accidental changes, and provide the ability to revert these changes before they lead to further problems.

Importance of monitoring Kubernetes object configuration

Monitoring Kubernetes object configuration is essential to ensure that the cluster remains in its desired state. There are several reasons why this is important:

1. Consistency: Ensuring that the cluster remains consistent with the desired configuration helps ensure that the applications running on the cluster remain stable and predictable.
2. Security: Monitoring configuration can help detect unauthorized changes, such as changes to network policies or other security-related objects, that could potentially expose the cluster to security risks.
3. Compliance: Monitoring configuration can help ensure that the cluster remains compliant with regulatory requirements, such as PCI-DSS or HIPAA.
4. Optimization: Monitoring configuration can help identify opportunities to optimize the cluster’s resource usage, such as identifying underutilized resources or identifying opportunities to scale applications more efficiently.
5. Disaster recovery: In the event of a disaster or outage, having an up-to-date and consistent backup of the Kubernetes configuration is critical for restoring the cluster to a stable state. By monitoring configuration changes, administrators can ensure that backups are up-to-date and accurate, reducing the risk of data loss and downtime.
6. Collaboration: In large organizations, it is not uncommon for multiple teams to manage different aspects of the Kubernetes environment. Monitoring Kubernetes configuration can help facilitate collaboration between these teams by providing a centralized view of the cluster’s configuration, making it easier to identify and resolve issues that may span multiple teams or applications.

Open-source tools for monitoring Kubernetes object configuration

Several open-source tools are available for monitoring Kubernetes object configuration. These tools help track how configurations such as configmaps and secrets have changed between deployments, monitor changes in the state of Kubernetes objects in the form of Kubernetes YAML spec, and ensure that the cluster remains in its desired state. Here are some of the current tools available for monitoring Kubernetes object configuration and events:

1. Kubernetes auditing: The Kubernetes auditing feature is a built-in capability that records all actions and events that occur within a Kubernetes cluster. This feature helps organizations maintain compliance, improve security, and troubleshoot issues by providing a detailed record of all activities within the cluster.
2. kube-diff: kube-diff is a command-line tool that can compare Kubernetes objects between two versions and show the differences. It can be used to track changes in configuration files and identify any unauthorized modifications.
3. Conftest: Conftest is a tool that can be used to test Kubernetes object configuration files against Open Policy Agent (OPA) policies. It can be used to ensure that the configuration files adhere to best practices and are consistent with the desired state of the cluster.
4. kubewatch: Kubewatch is a tool that can be used to monitor Kubernetes events and send notifications when changes occur. It can be used to monitor the state of Kubernetes objects and detect any unauthorized modifications.

These tools can be used to monitor changes to Kubernetes configurations and ensure that the cluster is operating according to its desired state. By tracking changes and detecting unauthorized modifications, administrators can take corrective actions before they lead to more significant issues.

With many Kubernetes monitoring tools available, companies can easily fall into tool sprawl, i.e., an excessive number of monitoring tools that are used to address different use cases, which can lead to complexity and inefficiency in the monitoring process. Using multiple tools also often means being limited to the monitoring and alerting capabilities of each tool, leading to the need to install a combination of tools and integrate them for full coverage. The installation and maintenance of monitoring tools, as well as the setup of integrations between them, can consume valuable development time that ideally should be spent on creating customer-focused features. Furthermore, these tools often offer one-dimensional insights into the infrastructure, requiring correlation with other data such as logs to obtain a comprehensive understanding of issues.

LogicMonitor’s container monitoring product, LM Container, consolidates Kubernetes object configuration monitoring with metrics, logs, and events generated by the application and the platform, allowing for a more holistic view of the system. This approach streamlines the monitoring process, improves visibility into the system, and reduces the complexity of managing multiple tools, making it a more efficient and effective solution for Kubernetes monitoring.

Monitoring Kubernetes object configuration and drift with LogicMonitor

At LogicMonitor, we are excited to announce new Kubernetes monitoring capabilities in LM Container that enable organizations to monitor changes in the configurations of Kubernetes objects. The new features in LM Container provide a comprehensive view of the current and past states of the Kubernetes objects and their configurations, enabling users to identify and address issues in real time.

One of the unique features of LM Container is the ability to store a backup copy (or gold copy) of the Kubernetes object configuration, which can be used to restore the Kubernetes object in case of a failure or outage. This allows users to quickly recover from any issues and ensure the availability of the application.

Another key aspect of LM Container is the ability to set alerts based on configuration value changes. Users can configure alerts to be triggered when a specific value in the configuration changes, such as the connection string to the database, enabling them to proactively detect and address issues before they become critical problems.

There is no learning curve when it comes to monitoring object configurations with LM Container. Use the LM Container installation guide to install the latest version of the LM Container Helm chart with default settings on your cluster, and LogicMonitor will take it from there.

Overall, the new Kubernetes monitoring features in LM Container are designed to provide a comprehensive view of the Kubernetes environment and ensure that it remains stable, secure, and efficient. With the ability to store gold copies of Kubernetes objects and set alerts based on configuration changes, businesses can ensure that their Kubernetes environment is well-monitored and optimized for maximum performance.

We will now examine some potential uses of LM Container for monitoring Kubernetes object configurations.

Common use cases

Ensuring stable Role-based Access Control (RBAC) policies

LM users can monitor changes to RBAC policies, such as cluster roles, cluster role bindings, role bindings, and others, to ensure only the appropriate users can access the cluster.

A cluster role binding policy is shown in the following screenshot. By default, any change to the policy will trigger a warning alert.

Users can monitor orphan roles and role bindings from the Maps/Topology view of the cluster as follows:

Ensuring compliance with enterprise policies on secrets

LM users can ensure the security and compliance of Kubernetes secrets configurations by monitoring changes to secret values.

Below is a screenshot showing a custom alert triggered when the password field is removed from the secret.

Ensuring consistent application settings and dependencies

LM users can track changes to Kubernetes configmaps to ensure consistent application settings and dependencies.

In the following screenshot, you can see an out-of-the-box alert triggered when there was a change to the configmap.

Identifying performance and availability issues with configuration monitoring

LM users can quickly identify configuration issues that may impact Kubernetes performance and availability, such as resource usage or network policies.

Simplifying disaster recovery

LM users maintain gold/backup copies of Kubernetes object configurations that can be quickly restored manually in the event of an issue.

Conclusion

Monitoring Kubernetes object configuration is crucial for ensuring the stability, security, and compliance of Kubernetes clusters. By monitoring changes to the cluster’s configuration, administrators can detect and address issues early on, preventing them from becoming critical problems. Additionally, monitoring configuration can help optimize resource usage and streamline the development process, leading to improved application performance and faster time-to-market.

There are several tools available for monitoring Kubernetes object configuration, ranging from basic command-line tools to more comprehensive and customizable solutions. Choosing the right tool will depend on the specific needs and requirements of your organization, as well as the complexity of your Kubernetes environment.

LogicMonitor offers new Kubernetes monitoring features that can monitor changes in configurations of Kubernetes objects, store a gold copy of the object configuration, and set alerts based on custom policies based on configuration values. LM Container enables organizations to quickly identify and address issues and ensure the availability and performance of their applications.

The benefits of monitoring Kubernetes object configuration cannot be overstated. It is a crucial part of maintaining a stable and secure Kubernetes cluster and can provide valuable insights into application behavior and resource usage. By prioritizing configuration monitoring, organizations can ensure that their Kubernetes environments are consistent, reliable, and optimized for maximum efficiency.