IN THIS ARTICLE:
- Introduction to Collector Capacity
- Device Capacity Limits
- NetFlow Capacity Limits
- Tuning Collector Size
Introduction to Collector Capacity
The amount of data that can be monitored by one Collector depends on the Collector's configuration and resources. You can monitor the load and capacity of your Collector using the Collector monitoring facilities of LogicMonitor. If you have a large environment, and are experiencing alerts on the Unavailable Task Rate datasource of your Collectors, you may need to tune your Collector to increase its monitoring capacity.
Device Capacity Limits
The following are representative measures of the capacity of differently sized Collectors of version GD 24.000 or later, measured in terms of a "standard device." The "standard device" assumes there are 50 instances collected by each protocol, checked with a 2 minute frequency. Additionally, the capacity is dependent on the number of instances that need to be discovered for each monitored device. For instance, if each device is a load balancer with 10,000 instances, the Collector capacity will be lower. Similarly, if each device is a switch with hundreds of interfaces, the Collector capacity may be lower because it is limited by discovery.
- If your Collector will be running on an Amazon EC2 instance, we recommend using a fixed performance instance type (e.g. M3, C3) instead of a credit based instance type (e.g. T2).
- While we strive to ensure that these measures are as accurate as possible, these are estimates and they may not be accurate for every production environment. These measures are subject to change.
- The Nano Collector size is not included in the following tables. This is due to the fact that the Nano Collector is intended for testing purposes. As such, no recommended device-count capacity has been assigned to it.
- Collectors using JDK 11 (28.400 and up) will see roughly 10% more memory and CPU usage than previous JDK 8 Collectors on the same hardware. We are working to improve this in future versions.
*Collector System Memory Requirements in a VM Environment
Part of a Collector's system memory allocation is devoted to SSE (Standalone Script Engine), which is enabled by default and used to execute script DataSources. In a non-VM environment, if there is not enough memory, the Collector won't initiate SSE and will instead execute script DataSources in the Agent. However, if the Collector is executed in a VM, because the OS indicates there is free memory, this safeguard can be overridden. This burst memory capacity in VMs can increase memory use above the system memory requirements listed previously. Although this can happen for Collector of any size, it is far more likely to happen to small Collectors.
To disable SSE and ensure additional memory is not used, update the groovy.script.runner configuration setting in the agent.conf file from sse to agent (e.g. groovy.script.runner=agent). If this setting is not present in your Collector's agent.conf file, look for the collector.script.asynchronous setting instead and update its value to false. For more information on updating a Collector's configuration files, see Editing the Collector Config Files.
NetFlow Capacity Limits
The following are representative measures of the flows per second that can be comfortably monitored by a Collector running in different sample environments:
Tuning Collector Size
You can adjust the Collector size directly from the LogicMonitor UI. This is particularly useful for performance tuning and increasing the capacity of your Collector after you have already installed it.
From Manage Collector | Support| Collector Configuration, select the desired Collector size from the dropdown menu. As soon as you select "Save and Restart," we will auto-verify that your host has enough available memory to support the new Collector size.
Please note the following:
- Older Collectors will display their current size as "Custom (xGB)" in the dropdown, even if no parameters have been modified since installing. This is because our definition of size has changed since the Collector was installed. If you want to ensure the Collector configuration is up to date, simply select the size you want (or had installed originally) and select Save and Restart.
- Changing a Collector's size has no effect on parameters unrelated to its size. The parameters listed in the section below, Configuration Details, are the only ones impacted by a change in the Collector's size.
If your Collector version is below EA 24.126, you will need to manually change the configuration files (wrapper, sbproxy, and agent) to match the specifications listed below for the size you desire. You can do so by selecting "Edit agent.config manually" and finding the necessary parameters in the text.
Minimum Recommended Disk Space for Collectors
Although the Collector operates in memory, certain operations- particularly caching- do require a small amount of available disk space on its host. The exact amount of required storage is highly variable and dependent of several factors such as Collector size, configuration, NetFlow usage, number of Collector logs, etc. Nevertheless, here are a few examples of required disk space based on these factors:
- A brand new install Collector will use about 500MB.
- At most, Collector logs will use 800MB.
- Temporary files (ie. upgrade files) will use less than 1500MB.
- Report cache data will use less than 500MB by default (this figure represents 30 minutes of cached data for a Large Collector)
- If using NetFlow the disk usage is less than 30GB.
In total, this means Collector disk usage will be, at most, less than 3.5GB without NetFlow and less than 33.5GB with NetFlow enabled.
Click the above link to view detailed information about the Collector's size and capacity