Open Source APM

Open Source APM 

This guide covers five popular open-source Application Performance Monitoring (APM) tools that DevOps teams and network administrators use to manage cloud applications in production. Each review examines features, functionality, and community adoption.

Application monitoring tools

For every website and mobile application, system logs are the central source of data on how the software operates on network hardware during production. Many automatically generated system logs from web servers are rough, unparsed, and hard to read. Application Performance Monitoring (APM) tools were developed by IT companies as a better way to display this data in historical charts for network analysis.

Application Performance Monitoring (APM) is a process for discovering, tracing, and diagnosing the behavior of cloud software applications in production. APM tools strengthen monitoring by providing deeper visibility into network topologies and improved metrics on the user experience for developers. One of the main advantages of APM tools is their ability to diagnose the root cause of common issues, such as slow UX loads or code errors.

As APM is a vast topic with many areas of specialization, the common characteristics are presented in the following table:

Open source APM tools

The APM market is well-developed, with many DevOps tools available from public cloud service providers, established IT vendors, and innovative startups. This list of five open-source APM tools examines projects commonly used in production for cloud application support, based on an analysis of their features and advantages. In this review, we selected:

1. Jaeger
2. Zipkin
3. Prometheus (with Grafana as user interface)
4. Elastic APM
5. Pinpoint

The sections below compare each tool across main features, system architecture, deployment complexity, recommended usage patterns, and UI interface. The ratings weigh ease of deployment, licensing cost, and production readiness.

Jaeger

Jaeger is an open-source, end-to-end distributed tracing tool, created by Uber in 2017. Jaeger is a CNCF-incubated project that monitors transactions in complex distributed systems. It’s a strong addition to any cloud runtime system based on a microservices architecture and is now part of Red Hat’s OpenShift Service Mesh, which integrates with both Kubernetes and OpenStack.

Teams use Jaeger in production to debug cloud software errors, monitor transactions, and optimize hardware performance.

Architecture

• Backend: Go
• UI: React/Javascript
• Storage: Cassandra 3.4+, Elasticsearch (5.x, 6.x, 7.x), Kafka
• Components: agent, collector, query, and an ingestor

Main features

• Distributed transactions monitoring – the ability to track down each transaction in the entire application stack (from the application to the database server).
• Performance and latency optimization – the ability to monitor system behavior during runtimes and catch bottlenecks.
• Root cause analysis – provides a deep dive into queries and systems that can help point out the issue.
• Service dependency analysis – the ability to visualize how system components are interconnected and how they communicate.
• Distributed context propagation – passing the metadata from various steps during the request executions to connect data from multiple checkpoints within the system.

Deployment

The main Jaeger components are the agent, collector, query, and ingestor. Each component has a specific role to play in Jaeger’s backend. The four main components of Jaeger are available as open-source code released as Docker images. For Kubernetes and OpenShift deployments, orchestrations are available on GitHub.

With these images and cloud orchestration, Jaeger’s deployment is relatively straightforward for experienced network administrators and data center engineers.

Jaeger can easily be deployed from your localhost by running this simple Docker command:

Once deployed, Jaeger can be accessed via a browser on http://:1668

User Interface

Jaeger has a simple UI that helps administrators track problematic transactions and profile them visually through hierarchies and charts.

Recommended Usage

Jaeger is relatively simple to integrate and lightweight in terms of requirements for processing on hardware. It’s recommended to include Jaeger in:

• Kubernetes Deployments
• Istio Deployments
• Microservices Architecture

Jaeger adds another dimension to your cloud service mesh and enables easier tracing of code errors while operating at an abstraction level that doesn’t interfere with running code or database queries.

Zipkin

Zipkin is a distributed tracing system for troubleshooting latency in service architectures. It was launched by Twitter in 2012. Zipkin includes an instrumentation library that allows developers to build reports via HTTP, Kafka, Apache ActiveMQ, gRPC, or RabbitMQ. There are also options for the persistent storage of metrics data for historical search and chart generation with Apache Cassandra or Elasticsearch.

Architecture

• Language: Java
• Storage: Cassandra, Elasticsearch, and MySQL
• 4 components: collector, storage, search, web UI

An instrumentation library needs to be created to send traces to Zipkin. There are different Zipkin libraries available for the most common cloud runtime environments. Teams typically send network event data via Kafka or via an HTTP API. The requirements for historical data storage will depend on the project.

Main features

• Collection and lookup of data – monitoring and visualizing application behavior.
• Dependency diagram – the ability to show how many requests went through each system layer.

Deployment

The easiest way to deploy Zipkin is by using a self-contained Zipkin server executable jar file:

Another option is to use a Docker image and run it by using the command:

Once deployed Zipkin will be available on http://:9411/zipkin/.

Jaeger/Zipkin comparison

Jaeger is built on top of Zipkin and adds a few important features to the base functionality. Jaeger is more aligned with the current CNCF ecosystem and has seen more recent releases and tooling updates. On the other hand, Zipkin is lightweight. It has multiple libraries for different platforms and can be used to build on top of them. Both solutions focus on distributed tracing and will yield similar results. Which one to choose depends on the particular details of your cloud software deployments.

Language support:	Community:	Container type of deployment:	Non-containerized deployment:
Jaeger	More official languages supported	More flexible, scalable, and performant
Zipkin		Has a larger developer community; More GitHub contributors, stars, and supporters	More mature and simple without heavy add-on requirements

User interface

Similar to Jaeger, Zipkin provides a simple UI interface for tracking down transactions. It also enables an advanced search based on trace IDs if applicable.

Prometheus

Prometheus was created in 2012 by SoundCloud and is CNCF’s second hosted project after Kubernetes, built for cloud system monitoring and alert automation. Prometheus collects and stores data as time series, enabling a wide variety of analytics add-on features.

Architecture

• Language: Go
• Storage: In memory and on the local disk in a custom format
• Visualization: Grafana or any other API consumer
• Components: Prometheus storage, client libraries, push gateway, exporters, alert manager

Main Features

• Dimensional data model – Ability to present the time series by metric name and key/pair values.
• Powerful queries – PromQL as a query language that helps to add dimensionality into the system.
• Great visualizations – Variety of graphics and dashboards to provide better data analytics displays.

Deployment

Prometheus can be deployed:

• Using pre-combined libraries available on the Prometheus downloads page
• From source by installing Go, Node.js, npm, and using the makefile available on the official GitHub account
• Using Ansible, Chef, Puppet, SaltStack, etc. as a configuration management solution from third-party contributors.
• Using Docker by building from images available on DockerHub via simple command:

docker run --name prometheus -d -p 127.0.0.1:9090:9090 prom/prometheus

After installation, Prometheus is available on http://:9090/.

Grafana visualization

Since the 2015 release of Grafana 2.5.0, Prometheus has featured Grafana’s built-in support for creating data visualizations. To integrate Grafana data charts and graphs using Prometheus, install Grafana following the official instructions.

Once installed, Grafana is available on: http://:3000/.

The next step is to add Prometheus as a data source and create a Prometheus graph. More details can be found on the Prometheus and Grafana documentation.