- About LogicMonitor
- Cloud Monitoring
- Dashboards and Widgets
- Getting Started
- LM Service Insight
- Backup and Recovery Systems
- Cloud Resources
- Networking & Firewalls
- Fortinet FortiWLC Monitoring
- Fortinet FortiWeb Monitoring
- Fortinet FortiSwitch Monitoring
- Fortinet FortiManager Monitoring
- Fortinet FortiMail Monitoring
- Fortinet FortiAuthenticator Monitoring
- Fortinet FortiADC Monitoring
- Ruckus ZoneDirector Monitoring
- Cisco VoIP Monitoring
- Cisco UCS Monitoring
- Cisco Wireless Monitoring
- Brocade Application Delivery Controllers
- Checkpoint Firewalls
- Cisco APIC Monitoring
- Cisco ASA/ASR
- Cisco Device SNMP and NTP Configuration
- Cisco Firepower Chassis Manager Monitoring
- Cisco IP SLA Monitoring
- Citrix NetScalers
- Dell Switch Monitoring
- F5 BIG-IP Monitoring
- Fortinet FortiGate Monitoring
- Infoblox Monitoring
- Interface Status alerting and Bandwidth Utilization
- Juniper SRX
- Kemp LoadMaster Load Balancers
- Meraki Cloud Access Controllers
- NetFlow Monitoring
- Palo Alto Firewalls
- pfSense Firewalls
- Sonicwall Firewalls
- Operating Systems & Virtualization
- VMware Horizon Monitoring
- Citrix XenServer Monitoring
- Citrix XenApp/XenDesktop Monitoring
- ESXi Servers and vCenter/vSphere Monitoring
- Linux Disk Performance
- Linux File Systems reporting more than 100% usage
- Linux Inodes
- Linux Interface Bandwidth Utilization
- Linux NFS Server
- Monitoring a Domain Controller (DC)
- Monitoring Remote Linux Files
- NTP Configuration
- NTP Monitoring
- Nutanix HyperConverged Infrastructure
- SNMP v1/v2 Configuration
- SNMPv3 Configuration
- Solaris Monitoring
- Troubleshooting Perfmon Access
- Troubleshooting SNMP
- Troubleshooting WMI
- VMware vCenter Server Appliance (VCSA) Monitoring
- Windows Cluster Monitoring
- Windows Firewall Issues
- Windows Server 2000
- Windows XP
- Applications & Databases
- Atlassian Statuspage (statuspage.io) Monitoring
- Microsoft Office 365 Monitoring
- OpenMetrics Monitoring
- Zoom Monitoring
- Windows Server Failover Cluster (on SQL Server) Monitoring
- Slack Status Monitoring
- Unomaly Monitoring
- Apache Monitoring
- Cassandra Monitoring
- ConnectWise Monitoring
- Email Service Monitoring
- Java Applications
- Lighttpd Monitoring
- Microsoft Exchange Monitoring
- Microsoft SQL Server Monitoring
- MongoDB Monitoring
- MySQL Monitoring
- Nginx Monitoring
- Oracle Monitoring
- Pick & D3
- Postfix Monitoring
- PostgreSQL Monitoring
- RabbitMQ Monitoring
- Redis Monitoring
- Twilio Monitoring
- Varnish HTTP Accelerator
- Server & Operations Hardware
- Storage Systems
- Cisco HyperFlex Monitoring
- Dell SC Monitoring
- Apache Hadoop Monitoring
- EMC ECS
- EMC Isilon Monitoring
- EMC Unity Monitoring
- EMC VMAX
- EMC VNX/Clariion SAN
- EMC VNXe
- EMC VPLEX
- EMC XtremIO
- HPE 3PAR Storage
- HP MSA / StorageWorks / P2000
- HP P4000/Lefthand SANs
- NetApp E/EF-Series Monitoring
- NetApp Monitoring
- Nimble Storage
- Panzura Cloud Storage
- Quantum Small Tape Libraries
- VMware vSAN Monitoring
- Rest API Developers Guide
- RPC API Developers Guide - Deprecated
- Servicenow CMDB Integration
- Terminology and Syntax
Apache Hadoop is a collection of software allowing distributed processing of large data sets across clusters of commodity hardware. The LogicMonitor Hadoop package monitors metrics for the following components:
- HDFS NameNode
- HDFS DataNode
As of February 2020, we have confirmed that our Hadoop package is compatible with version 3.2.1. It may be possible to monitor older versions of Hadoop, but data will not be returned for all datapoints.
As Apache releases newer versions of Hadoop, LogicMonitor will test and extend coverage as necessary.
Enable JMX on Hadoop Host
LogicMonitor collects Hadoop metrics via the REST API rather than directly via JMX. However, metrics are originally collected and stored using JMX and, therefore, JMX must be enabled on the Hadoop host. For more information on enabling JMX, see the “Enabling JMX” section of the Java Applications (via JMX) Monitoring support article.
Add Hosts Into Monitoring
Add your Hadoop host(s) into monitoring. For more information on adding resources into monitoring, see Adding Devices.
Assign Properties to Hadoop Resources
The following custom properties must be set on the Hadoop resource(s) within LogicMonitor. For more information on setting properties, see Resource and Instance Properties.
Note: These ports must be open to the Collector.
Note: To verify the correct port is being used, you should be able to access http://<HOST>:<HTTP_PORT>/jmx and view metrics for each of the various components.
From the LogicMonitor repository, import all Hadoop LogicModules, which are listed in the LogicModules in Package section of this support article. Upon import, these LogicModules will be automatically associated with your Hadoop resources, assuming the properties listed in the previous section are assigned.
LogicModules in Package
LogicMonitor’s package for Apache Hadoop consists of the following LogicModules. For full coverage, please ensure that all of these LogicModules are imported into your LogicMonitor platform.
Configuring Datapoint Thresholds
The Hadoop package does not include predefined datapoint thresholds (in other words, no alerts will trigger based on collected data). This is because the technology owner has not provided KPIs that can be reliably extended to the majority of users. In order to receive alerts for collected data, you’ll need to manually create custom thresholds, as discussed in Tuning Static Thresholds for Datapoints.
Next are some datapoints for which you may want to consider setting thresholds:
- DataSource: Hadoop HDFS DataNode FS State
- NumFailedVolumes. Datapoint that reports total number of failed volumes.
- Remaining. Datapoint that reports remaining capacity on the datanode.
- DataSource: Hadoop HDFS NameNode Info
- NumberOfMissingBlocksWithReplicationFactorOne. Datapoint that reports the number of blocks with only one copy across the cluster.
- PercentUsed. Datapoint that reports the percentage of used space across the cluster (DFS and non-DFS).
- DataSource: Hadoop HDFS NameNode Status
- ServiceRestart. Datapoint that returns a value greater than 0 when the service state changes
- State. Datapoint that returns a status code indicating the status of the Hadoop namenode service.
- DataSource: Hadoop HDFS NameNode FSNamesystem
- CorruptBlocks. Datapoint that reports the current number of blocks with corrupt replicas.
- CorruptReplicatedBlocks. Datapoint that reports the number of corrupt blocks that have been replicated.
- FSState. Datapoint that returns a status code indicating whether the FS is operational or in safe mode.
- MissingBlocks. Datapoint that reports the current number of missing blocks.
- MissingReplicationOneBlocks. Datapoint that reports the number of missing blocks with replication factor of 1.
- NumDeadDataNodes. Datapoint that reports the number of datanodes currently dead.
- UnderReplicatedBlocks. Datapoint that reports the current number of blocks under replicated.
- VolumeFailuresTotal. Datapoint that reports the total number of volume failures across all datanodes.
- DataSource: Hadoop Yarn Queue Metrics
- AppsFailed. Datapoint that reports the number of applications that failed to complete.
- DataSource: Hadoop Yarn Cluster Status
- NumLostNMs. Datapoint that reports the current number of lost NodeManagers for not sending heartbeats.
In this Article: