Monitoring Microservices

Monitoring microservices

Software architecture has changed significantly over the last decade, from traditional, tightly coupled monolithic applications to loosely coupled microservices. Software monitoring has had to adapt to suit this new architecture. Monitoring microservice health isn’t as simple as tracking the CPU and memory utilization of a few servers. In this article, you’ll learn how to effectively monitor microservices using a standard set of alarms paired with effective logging and tracing.

Traditional vs. microservice monitoring

Microservices are small, single-purpose application components that together form a complete software application. These services typically run inside containers (such as Docker) and scale across multiple hosting environments.

The application architecture based on microservices is designed for high performance, high availability, efficient use of computing resources, and faster development. Applications built this way handle variable workloads and failures gracefully by running many small containers. Additionally, development is safer and faster when working on a single microservice at a time.

Microservice architectures also introduce significant operational complexity that exceeds legacy architectures. Applications have tens or even hundreds of unique microservices. SSH access to individual containers doesn’t scale in a microservice architecture.

Containers are an additional layer of abstraction on top of Virtual Machines (VMs), creating distinct infrastructure metrics and logs. Many versions of the same application run inside containers across multiple physical locations. The lifespan of containers is shorter than that of VMs, meaning logs and metrics are harder to access during debugging.

Microservice monitoring challenges

Below is a table that describes monitoring goals, their traditional solutions, and the challenges you’ll face when moving to microservices.

Traditional infrastructure monitoring doesn’t map cleanly to microservice environments. The range of infrastructure that needs monitoring is wide: VMs, containers, autoscaling groups, load balancers, Kubernetes pods, and database servers. But not all of it requires alarms.

The solution for effective microservice monitoring is understanding the difference between alarming metrics and debugging metrics. We’ll then add logs and transaction tracing on top of this solid metric foundation.

Alarming metrics vs. debugging metrics

Alarms are notifications sent to on-call engineers indicating a problem with the system. We refer to the key metrics that trigger these alarms as alarming metrics. In contrast, engineers use debugging metrics to identify the root cause of the alarm.

The critical difference between alarming and debugging metrics is that alarming metrics must reflect user-facing symptoms. Alarms notify engineers that they must intervene to fix the system. If a system user isn’t experiencing symptoms, then the problem doesn’t require human intervention. Examples of user-facing symptoms are:

• Slow interactions with the user interface
• Errors when using a public API
• Missing data or notifications
• Delays in batch processing

Debugging metrics are detailed application-level metrics. Only Subject Matter Experts (SMEs) need to understand these details for an application. Debugging metrics are helpful during the investigation of an incident but are overly specific as alarm triggers. Overly granular metrics can confuse engineers and slow the remedial response.

Four key alarming metrics

The table below shows four metrics applicable to all microservices. The two dimensions are:

• The requests that come from users’ browsers via HTTP and the requests that come from an application microservice via an application message bus or an Application Programming Interface (API).
• The error percentages and delay times as measured for those requests.

HTTP requests are calls by a user or other application to your application. Examples of requests are a user logging in, the user interface requesting data for a dashboard, or another application sending metric data to your application. HTTP requests are synchronous because your application must respond immediately. The microservices communicate with each other via an application message bus and/or an API to collectively process a user transaction. The message bus and/or API are ideal points for collecting metrics related to the availability, performance, and error-free operations of the microservices (we’ve devoted this article to help readers understand the best practices associated with API monitoring).

Alternatively, some work within your application is processed asynchronously. Batch jobs, nightly email reports, or backend billing calculations don’t run in response to real-time requests. Message buses (such as Kafka) store messages related to asynchronous tasks, and services pull work from the bus when they’re ready.

Pushed versus pulled work is an easy way to distinguish between synchronous and asynchronous requests. Synchronous requests push requests onto services. Services pull asynchronous requests when they complete their current tasks. Both types of requests can result in errors and delays.

Your system, subsystems, and microservices each need this type of monitoring. The following section describes the advantages of this three-level monitoring.

Three-level monitoring

The metrics described above treat the monitored system as a black box. Once your application grows beyond a few microservices, the implementation details become too many for a single engineer to understand. The four metrics above indicate user-facing symptoms since errors and delays are the primary symptoms users experience during an outage or a slowdown, regardless of the inner complexity of each microservice.

Alarming on errors and delays across the whole system is a significant first step, but it isn’t an effective microservice monitoring strategy. Monitoring only the whole system will detect total system outages. With microservice architectures, subsystems and services are decoupled from one another, reducing the risk of total system outages. Decoupling services increases the risk of partial system outages, though. You must implement alarms at the subsystem and service level to monitor these partial system outages.

The advantage of this three-level monitoring approach is that alarms immediately indicate the subsystem or microservice affected by the incident. On-call engineers can escalate the incident to subject matter experts (SMEs) for the affected service early in the debugging process, thereby shortening the incident duration.

Three-level monitoring examples

To highlight how three-level monitoring works, consider the following examples.

Imagine a system that ingests data, processes that data asynchronously using a message bus, and then sends email notifications back to the customer. There are three subsystems in this system: data ingestion, data processing, and notification sending.

If you receive an HTTP error alarm for data ingestion, the subsystem may require human intervention. The reasons behind the user-facing errors are varied. For example, a subsystem may use a cache that’s failing, a slow database, or rely on containers that are producing errors. Such a scenario requires an alarm so that an expert can diagnose the specific issue using debugging metrics.

Alternatively, focusing on debugging metrics can cause false alarms. The notification subsystem in our example may rely upon a third-party email provider. Third-party provider failure count is important to track, but it isn’t a reliable alarming metric.

Consider what happens when the third-party provider has a partial outage. The third-party provider failure count metric will jump. If your team has prepared for this scenario, the notification subsystem will automatically postpone sending the delayed emails upon encountering this error. In other words, there are no user-facing symptoms and, therefore, no need for manual intervention.

Layering in infrastructure metrics

We can now return to traditional monitoring metrics, since we understand the difference between alarming and debugging metrics. CPU percentage, memory utilization, and other server-level metrics are no longer vital alarming metrics when monitoring microservices, but they’re valuable debugging metrics. Infrastructure metrics, along with detailed application metrics, help SMEs determine an incident’s root cause after isolating the part of the application infrastructure that contributed to it.

Layering in logs and tracing

This article has focused on application and server metrics because they are key to proper alerting. Metric values are aggregated to summarize information for operators, but this also limits detailed debugging. Aggregation strips metrics of their context. For example, an error count indicates that a service has errors, but doesn’t describe the error messages. Centralized logging (collecting and visualizing all logs in a tool such as an Elasticsearch, Logstash, and Kibana stack) and transaction tracing (e.g., Jaeger) are used to apply context to aggregated metrics.

As mentioned earlier in this article, microservices break application logic into multiple loosely-coupled services. Application logs and tracing tie microservices back together, allowing operators to pinpoint errors as they propagate through the system. When combined with infrastructure monitoring and Internet path visibility, logs and traces help teams close blind spots from the end user all the way down to the underlying infrastructure. Logging tools like Elastic collect text-based log files from distributed systems and index them to help users isolate specific error messages by searching. Tracing tools like Jaeger discover and visualize relationships between microservices, helping users understand dependencies and isolate root causes.

Publishing too many logs or traces can become expensive, though. Consider sampling logs and traces to reduce storage costs. You can also reduce your storage costs by implementing dynamic log levels, allowing operators to enable or disable detailed logging in a production environment at runtime.

Conclusion

Effective microservice monitoring starts with the end-user experience and extends to every supporting service. Monitoring remains foundational, but modern observability practices, such as centralized logging, distributed tracing, and AI-assisted correlation, strengthen it by connecting end-user symptoms to their root causes across infrastructure, network, and cloud layers.

The challenge in monitoring a microservice-based application lies in the volume of data it generates and the volatility of its infrastructure. Servers, containers, load balancers, databases, and other system components are short-lived and continuously produce time-series data and log files.

An effective approach to monitoring a microservice is to treat it as a black box and rely on the service latency and error rate to mask its inner complexity. These two metrics are well-suited for alarming because they’re simple to understand and apply to a microservice as they apply to an entire software application.

The alarms generated by these two metrics can be forwarded to experts who specialize in a given microservice to isolate the root cause of the problem. The experts must rely on distributed tracing to determine where the problem originated and use log monitoring tools to search for the earliest indicative error message. Unified platforms that bring metrics, logs, and traces together and correlate them with Internet path and infrastructure data reduce the time spent switching between tools.

Monitoring microservices improves incrementally. After each outage, refine alarm thresholds and review incidents for patterns. Add metrics and log entries that make future troubleshooting more efficient.

FAQs