Support Center Home


Collector Capacity

The amount of data that a Collector can handle depends on the Collector’s configuration and resources. You can monitor the data collection load and performance of your Collector to minimize disruption and notify when a collector is down. See Monitoring your Collectors.

If you have a large environment, and are experiencing alerts on the Unavailable Task Rate datasource of your Collectors, you may need to tune your Collector to increase its monitoring capacity.

Device Capacity Limits

The following table describes the capacity of differently sized Collectors measured in requests per second (RPS) and in terms of a “standard device”, which assumes there are 50 instances collected by each protocol checked with a 2 minute frequency. These measurements are estimates and may not be accurate for every production environment.

Protocol Small Collector Medium Collector Large Collector Extra Large (XL) Collector Double Extra Large (XXL) Collector
CPU:1 Intel Xeon Family

System Memory: 2GiB

JVM maximum memory: 1GiB
CPU:2 Intel Xeon E5-2680v2 2.8GHz

System Memory: 4GiB

JVM maximum memory: 2GiB
CPU:4 Intel Xeon E5-2680v2 2.8GHz

System Memory: 8GiB

JVM maximum memory: 4GiB
CPU: 8

System Memory: 16GiB

JVM maximum memory: 8GiB
CPU: 16

System Memory: 32GiB

JVM maximum memory: 16GiB
SNMP v2c 500 RPS
(1200 standard devices)
750 RPS
(1800 standard devices)
1250 RPS
(3000 standard devices)
1660 RPS
(4000 standard devices)
1880 RPS
(4500 standard devices)
SNMP v3 500 RPS
(1200 standard devices)
666 RPS
(1600 standard devices)
833 RPS
(2000 standard devices)
1000 RPS
(2400 standard devices)
1170 RPS
(2800 standard devices)
HTTP 160 RPS
(320 standard devices)
735 RPS
(1400 standard devices)
1260 RPS
(2400 standard devices)
2000 RPS
(4500 standard devices)
3740 RPS
(7500 standard devices)
WMI 82 RPS
(160 standard devices)
620 RPS
(1200 standard devices)
1033 RPS
(2000 standard devices)
TBA TBA
Perfmon 87 RPS
(200 standard devices)
173 RPS
(400 standard devices)
347 RPS
(800 standard devices)
TBA TBA
JMX 416 RPS
(1000 standard devices)
1041 RPS
(2500 standard devices)
2083 RPS
(5000 standard devices)
TBA TBA

The capacity also depends on the number of instances that need to be discovered for each monitored device. For example, if each device is a load balancer with 10,000 instances, Collector capacity will be lower. And if each device is a switch with hundreds of interfaces, Collector capacity may be lower because it is limited by discovery.

Notes:

  • If your Collector will be running on an Amazon EC2 instance, we recommend using a fixed performance instance type (such as M3 or C3) instead of a credit based instance type (such as T2).
  • The Nano Collector size is not included in the following tables. This is due to the fact that the Nano Collector is intended for testing purposes. As such, no recommended device-count capacity has been assigned to it.
  • Collectors using JDK 11 (supported by Collector version 28.400 and higher numbered versions) will see roughly 10% more memory and CPU usage than previous JDK 8 Collectors on the same hardware. We are working to improve this in future versions.

Collector memory requirements in a VM

Part of a Collector’s system memory allocation is devoted to Standalone Script Engine (SSE), which is enabled by default and used to execute script DataSources (Groovy scripts).

Collector Size SSE Memory Requirements
Small 0.5GiB
Medium 1GiB
Large 2GiB
Extra Large 4GiB
Double Extra Large 8GiB

In general, the SSE requires half of the amount of memory allotted to the JVM. The memory requirements are not shared, rather the SSE requirement is in addition to the JVM memory requirements. If the Collector does not have this memory available, the SSE will not start and you will see “Can’t find the SSE Collector Group” in the Collector Status dialog. The Collector will work without the SSE, but Groovy scripts will be executed from the Agent instead of the SSE.

If the Collector is executed in a VM, this safeguard can be overridden because the OS indicates there is free memory. This burst memory capacity in VMs can increase memory use above the system memory requirements listed previously. Although this can happen for Collector of any size, it is far more likely to happen to small Collectors.

To disable SSE and prevent additional memory use, edit the Collector’s agent.conf:

  • If the configuration setting reads groovy.script.runner=sse, change it to groovy.script.runner=agent.
  • If the previous setting is not present, update the following setting: collector.script.asynchronous=false.

For more information, see Editing the Collector configuration files.

NetFlow Capacity

Using the sample environments called out in the previous capacity table, the following are measurements of the Network Traffic Flow (NetFlow) per second that can be monitored by the various-sized Collectors.

Linux 64-bit

Collector Size No. of Monitored Devices Supported Flows/Sec
Small 1 TBA
5 TBA
Medium 1 TBA
5 TBA
Large 1 113380
5 92740
XL 1 164870
5 117730
XXL 1 187870
5 130530

Windows 64-bit

Collector Size No. of Monitored Devices Supported Flows/Sec
Small 1 TBA
5 TBA
Medium 1 TBA
5 TBA
Large 1 92040
5 72950
XL 1 120120
5 74890
XXL 1 120540
5 79420

Tuning Collector size

You can adjust the Collector size from the LogicMonitor UI, especially for performance tuning and increasing the Collector capacity after you installed it.

From Manage Collector | Support| Collector Configuration, select the Collector size from the dropdown menu. After you “Save and Restart” the settings, LogicMonitor will automatically verify that your host has enough available memory to support the new Collector size.

Note:

  • Older Collectors will display their current size as “Custom (xGiB)” in the dropdown, even if no parameters have been modified since installing. This is because our definition of size has changed since the Collector was installed. If you want to ensure the Collector configuration is up to date, simply select the size you want (or had installed originally) and select Save and Restart.
  • Changing a Collector’s size has no effect on parameters unrelated to its size. The parameters listed in the section below, Configuration Details, are the only ones impacted by a change in the Collector’s size.

If you are manually changing the Collector’s config parameters, we will run a validity check after you select “Save and Restart” to ensure that no errors were made in the new configuration. If errors are detected, we will display which lines are missing/duplicated so they can be corrected.

Configuration Details

Small Collector

Config File Parameters Description
wrapper.conf wrapper.java.initmemory=128 Minimum Java Heap Size(MiB) for Collector
wrapper.java.maxmemory=1024 Maximum Java Heap Size(MiB) for Collector
sbproxy.conf wmi.stage.threadpool.maxsize=100 The maximum size of threads to handle WMI query/fetch data in sbwinproxy.exe
wmi.connection.threadpool.maxsize=50 The maximum size of threads for WMI to connect to remote machine in sbwinproxy.exe
agent.conf sbproxy.connector.capacity=8192 The maximum number of requests that the Collector can send in parallel to sbwinproxy and sblinuxproxy
discover.workers=10 Allocates resources to Active Discovery iterations
autoprops.workers=10 The thread pool size for AP
reporter.persistent.queue.consume.rate=10 The max count of data entries that will be reported for each API call.
reporter.persistent.queue.consumer=10 The thread count used to read from buffer and execute reporting.
collector.script.threadpool=100 The max thread count to run script tasks.
website.conf sse.max.spawn.process.count=3 N/A

Medium Collector

Config File Parameters Description
wrapper.conf wrapper.java.initmemory=512 Minimum Java Heap Size(MiB) for Collector
wrapper.java.maxmemory=2048 Maximum Java Heap Size(MiB) for Collector
sbproxy.conf wmi.stage.threadpool.maxsize=200 The maximum size of threads to handle WMI query/fetch data in sbwinproxy.exe
wmi.connection.threadpool.maxsize=100 The maximum size of threads for WMI to connect to remote machine in sbwinproxy.exe
agent.conf sbproxy.connector.capacity=8192 The maximum number of requests that the Collector can send in parallel to sbwinproxy and sblinuxproxy
discover.workers=40 Allocates resources to Active Discovery iterations
autoprops.workers=10 The thread pool size for AP
reporter.persistent.queue.consume.rate=12 The max count of data entries that will be reported for each API call.
reporter.persistent.queue.consumer=10 The thread count used to read from buffer and execute reporting.
collector.script.threadpool=200 The max thread count to run script tasks.
website.conf sse.max.spawn.process.count=5 N/A

Large Collector

Config File Parameters Description
wrapper.conf wrapper.java.initmemory=1024 Minimum Java Heap Size(MiB) for Collector
wrapper.java.maxmemory=4096 Maximum Java Heap Size(MiB) for Collector
sbproxy.conf wmi.stage.threadpool.maxsize=400 The maximum size of threads to handle WMI query/fetch data in sbwinproxy.exe
wmi.connection.threadpool.maxsize=200 The maximum size of threads for WMI to connect to remote machine in sbwinproxy.exe
agent.conf sbproxy.connector.capacity=16384 The maximum number of requests that the Collector can send in parallel to sbwinproxy and sblinuxproxy
discover.workers=80 Allocates resources to Active Discovery iterations
autoprops.workers=15 The thread pool size for AP
reporter.persistent.queue.consume.rate=12 The max count of data entries that will be reported for each API call.
reporter.persistent.queue.consumer=15 The thread count used to read from buffer and execute reporting.
collector.script.threadpool=300 The max thread count to run script tasks.
website.conf sse.max.spawn.process.count=5 N/A

XL Collector

Config File Parameters Description
wrapper.conf wrapper.java.initmemory=1024 Minimum Java Heap Size(MiB) for Collector
wrapper.java.maxmemory=8192 Maximum Java Heap Size(MiB) for Collector
sbproxy.conf wmi.stage.threadpool.maxsize=800 The maximum size of threads to handle WMI query/fetch data in sbwinproxy.exe
wmi.connection.threadpool.maxsize=400 The maximum size of threads for WMI to connect to remote machine in sbwinproxy.exe
agent.conf sbproxy.connector.capacity=32768 The maximum number of requests that the Collector can send in parallel to sbwinproxy and sblinuxproxy
discover.workers=160 Allocates resources to Active Discovery iterations
autoprops.workers=20 The thread pool size for AP
reporter.persistent.queue.consume.rate=15 The max count of data entries that will be reported for each API call.
reporter.persistent.queue.consumer=20 The thread count used to read from buffer and execute reporting.
collector.script.threadpool=400 The max thread count to run script tasks.
website.conf sse.max.spawn.process.count=10 N/A

XXL Collector

Config File Parameters Description
wrapper.conf wrapper.java.initmemory=2048 Minimum Java Heap Size(MiB) for Collector
wrapper.java.maxmemory=16384 Maximum Java Heap Size(MiB) for Collector
sbproxy.conf wmi.stage.threadpool.maxsize=1600 The maximum size of threads to handle WMI query/fetch data in sbwinproxy.exe
wmi.connection.threadpool.maxsize=800 The maximum size of threads for WMI to connect to remote machine in sbwinproxy.exe
agent.conf sbproxy.connector.capacity=65536 The maximum number of requests that the Collector can send in parallel to sbwinproxy and sblinuxproxy
discover.workers=320 Allocates resources to Active Discovery iterations
autoprops.workers=30 The thread pool size for AP
reporter.persistent.queue.consume.rate=20 The max count of data entries that will be reported for each API call.
reporter.persistent.queue.consumer=30 The thread count used to read from buffer and execute reporting.
collector.script.threadpool=600 The max thread count to run script tasks.
website.conf sse.max.spawn.process.count=15 N/A

Minimum Recommended Disk Space

Although the Collector operates in memory, operations such as caching require available disk space on its host. The exact amount of required storage varies and depends on factors such as Collector size, configuration, NetFlow usage, number of Collector logs, and so on.

These are examples of required disk space based on these factors:

  • A brand new install Collector will use about 500MiB.
  • At most, Collector logs will use 800MiB.
  • Temporary files (ie. upgrade files) will use less than 1500MiB.
  • Report cache data will use less than 500MiB by default (this figure represents 30 minutes of cached data for a Large Collector)
  • If using NetFlow the disk usage is less than 30GiB.

In total, this means Collector disk usage will be less than 3.5GiB without NetFlow and up to 33.5GiB with NetFlow enabled.

In This Article