LogicMonitor has simplified creating and setting up OpenMetrics (Prometheus Exposition Format) DataSources for polling an endpoint where these metrics are exposed. The scripted DataSource templates are still available, but the Collector now supports the OpenMetrics data format, so you no longer need to use a script to parse the data. The recommended way to create new DataSources is to use the wizard.
The wizard is found in your LogicMonitor portal in the Exchange page. Navigate to the Exchange page and select OpenMetrics from the menu tabs. The OpenMetrics tab shows all the DataSources that have been created in the account.
- EA Collector 30.100 or later installed and monitoring the resource from an OpenMetrics endpoint.
- OpenMetrics host(s) added into monitoring. For more information about adding resources into monitoring, see Adding Devices.
The OpenMetrics format is 100% compatible with the Prometheus Exposition Format on which it is based and has been adopted as the official standard by the Cloud Native Computing foundation.
The setup wizard allows you to load metrics from an OpenMetrics endpoint and select the metrics to process into datapoints for collection. At this time, the host resources need to be in monitoring and a new property for the endpoint URL must be added to the resource.
Set properties on host resource
Set the following custom properties on the OpenMetrics host(s) within LogicMonitor. For more information on setting properties, see Resource and Instance Properties.
|openmetrics.url||(Required) If the endpoint does not use basic authentication or HTTPS, this is the only property required to get started.
|openmetrics.host||(Required if not using openmetrics.url) To use parameterized values for openmetrics.url, you need to supply a host, port, and metrics if overriding defaults listed below.|
|openmetrics.headers||(Optional) Specify header values in the format:
|openmetrics.pass||(Optional) Password for authentication.|
|openmetrics.path||(Optional) Populate to override default path, ‘/metrics’.|
|openmetrics.port||(Optional) Populate to override default port, ‘9273’.|
|openmetrics.redirect||(Optional) By default is set to False.|
|openmetrics.ssl||(Optional) By default is set to False, which means HTTP.|
|openmetrics.timeout||(Optional) By default is set to 2000.|
|openmetrics.user||(Optional) Username for authentication.|
If you’ve ever created or edited a DataSource, these fields are the same as the General Information section for any other DataSource to include identifying information.
The unique name of the DataSource. This field is required and represents the name of the DataSource for how it will be displayed in the OpenMetrics DataSource list as well as in the Settings | LogicModules | DataSources list for viewing and editing.
As a best practice, this name should be descriptive. For example, you can specify the platform or application name first, and then specify a specific component of the platform.
The name of the DataSource as it will appear on the Devices page. This name does not need to be unique, for example there can be several DataSources that display as “CPU” and apply to different kinds of devices.
As a best practice, the display name should be shorter than the full DataSource name. For example, use “Services” instead of “Windows Services”, or “VIPs” instead of “Citrix Netscaler VIPs”.
Note: Names and display names for DataSources cannot include the operators and comparison functions because they are reserved for LogicMonitor’s datapoint expression syntax. See Complex Datapoints.
Descriptive information that will be associated with this DataSource. If the DataSource name is not enough to describe what the DataSource is collecting, then the Description field should provide enough information that someone seeing the name and description will understand what the DataSource does.
This field can contain any technical notes associated with the DataSource.
The DataSource Group to which the DataSource will be added. If this field is empty the DataSource will not be added to a DataSource Group. If you enter text that does not match an existing DataSource Group, one will be created.
For example, all Dell hardware monitoring DataSources are assigned the group Dell. This collapses those DataSources into a single DataSource group entry that can be expanded.
Use an AppliestTo script to define which resources to associate with this DataSource. See AppliesTo Scripting Overview.
After you define the General Information for the DataSource, the next step is to Load Metrics. To reduce the amount of setup required in this wizard, some fields default to common values which can be updated when editing the DataSource.
Select a Resource from the list. The Resource should have an endpoint defined for the metrics that your DataSource will collect. To show up in this list a resource has to have one of the following properties defined (see Setup Requirements):
|openmetrics.url||The full URL where the metrics are exposed and accessible by the Collector for the given Resource.|
|openmetrics.host||The hostname or IP address of the Resource where the OpenMetrics endpoint is exposed.
Note: This property uses the default values of port (9273) and path (/metrics) if their properties are not specified.
Once the metrics from the selected Resource have been loaded in the table, select a set of metrics for which you want to define datapoints for your OpenMetrics DataSource.
- You can select metric labels and exact or regex values to provide instance filters for Active Discovery. For example, metrics may be available for multiple environments and tagged with a label (such as test, dev, prod). This filter can be used to only discover instances from one of the environments, such as the “prod” environment.
- You can select metrics using a split wildcard search, by using an * to separate the keywords to match. For example, searching for query*slice*quantile will return metrics that contain “query”, “slice”, and “quantile” regardless of the text between those keywords.
- The Metric Name Filter allows you to limit the number of metric that are returned from an endpoint. Only those metrics that start with the test entered in the input box will be returned. This filter may be helpful for endpoints that expose a large number of metrics which may cause a longer time to load those metrics.
Note: If you are defining a multi-instance DataSource (such as a Resource that contains multiple CPUs, Disks, Interfaces, Processes, and so on) the labels which define those instances must exist across all selected metrics. Error checking is provided in the Add DataPoints step to ensure that instance definition is consistent.
After you have selected the metrics for your DataSource, click Next to “Add Instances”.
If the metrics you selected in the previous step do not have any labels defined, you can skip this step and continue to “Add DataPoints”.
Select a label that exists across all the selected metrics which defines the various instance names for the metric. This could be a label for CPU, Disk, Interface, Process, or Job. Each value of this label represents a new and separate instance from which to collect metrics.
Test instance settings
If you have defined an Instance label, you can use the Test Instance Settings button to display the instance values determined by the selected label. This is similar to the Test Auto-Discover option when viewing DataSources.
Enable auto discovery
(Recommended) Because of the dynamic nature of the OpenMetrics data, new instances may be added at any time. Enabling this feature means new instances will be discovered and added to LogicMonitory automatically.
Instance group label
Select any remaining label as an instance group. If you are gathering metrics from Test, QA, and Prod processes, you can group the instances by an “environment” label so that when you view them in the Resource tree, they will be organized by the environment.
Note: Adding a large number of DataPoints for a single DataSource can cause the data to display in the Resources tree incorrectly and can cause performance issues, such as high CPU usage for the collector. LogicMonitor recommends creating DataSources based on their purpose or function as opposed to dumping a large number of DataPoints of various types into a single DataSource. For example, you should create separate DataSources for CPU, garbage collection, error/success counts, and latency.
Based on the instance labels you selected and if there are no other labels, those metrics with matching metric name and instance label will be collapsed into a single metric. If these DataPoint definitions are correct, you can click Finish.
To edit any of the DataPoint definitions, click the edit pencil. You can:
- Change the DataPoint Name and Description.
- Aggregate metrics into a single DataPoint.
- Filter out metrics based on label values to only capture the values you care about.
To filter out metrics, select a label, operator, and value to match. Then select an aggregation type: sum or avg.
|Equals||You can use the Equals operator to filter for a specific value of the label. For example, if you have an “environment” label, you can select “prod” to collect data only from the Prod environment but not for Test and QA.|
|RegEx||For the RegEx operator, you can use regular expression and wildcards to match multiple metrics and aggregate the values of those metrics. You can enter * to filter for any value, or ^foo to filter for all values that start with foo.
For example, if you have labels that represent a reason for a failure, and you want the DataPoint to capture the total number of failures, your filter may include: label=reason, operator=equals, value=*, and aggregation method=sum.
When you are done editing your DataPoints, save your changes and then click the Finish button.
Note: If the Finish button is disabled, you may have improperly defined DataPoints. Check the DataPoint list to make sure the status for all is green. You can hover over the red status to see more information about the error. A common error may be that the metric does not contain the instance label you selected. In this case, either go back and select the correct instance label or delete the datapoint to proceed.
From the Finish screen you can select to create another DataSource or return to the OpenMetrics page. Creating a new DataSource will start the process again.