AIOps

Topology Mapping Overview

Introduction to Topology Mapping

FEATURE AVAILABILITY: LogicMonitor Pro and Enterprise

Topology mapping is the visual representation of relationships among elements within a communications network. Topology maps can represent the physical location of network components, generally referred to as layer 1 mapping, or they can represent the logical relationships among elements, referred to as layer 2 mapping.

LogicMonitor’s topology mapping capabilities are focused on layer 2 mapping. Specifically, the LogicMonitor platform leverages the Link Layer Discovery Protocol (LLDP) as well as Cisco’s proprietary version of the protocol known as Cisco Discovery Protocol (CDP) to dynamically generate network topology maps that show how data flows among the many resources (e.g. switches, hosts, firewalls, routers, and other network components) in your environment.

LogicMonitor takes an additive approach to generating topology maps. This approach initially shows only the relationships that are most relevant for the current task at hand, but can be expanded outward. For more information on creating, saving, and adding topology maps to your dashboards, see Mapping Page.

The visual context that LogicMonitor’s topology maps provide can be very beneficial to your monitoring operations, providing you with the ability to:

  • Visualize and navigate resources based on topology relationships
  • Troubleshoot alerts within your infrastructure on a topology map
  • Generate topology maps based on alerts to streamline your troubleshooting workflow
  • Discover and map relationships between resources
  • Save relevant and frequently-used topology maps for monitoring continuity

Getting Started

In order to enable topology mapping, you’ll need to import new and updated LogicModules as well as confirm you’re running the required Collector release, as outlined next:

  1. From the LogicMonitor repository, import all files responsible for assigning external resource IDs (ERIs) and external resource types (ERTs) to the resources currently monitored by LogicMonitor. As discussed in the External Resource IDs (ERIs) and External Resource Types (ERTs) section of this support article, ERIs are properties set on devices, instances, and services that serve to uniquely identify a resource.

    • To assign device-level ERI and ERT properties, please import all PropertySources prefixed with “addERI.” For example: addERI_Cisco, addERI_Vcenter, and so on. 

      1. You will also need to import the PropertySource addCategory_TopoSwitch. This helps LogicMonitor identify those resources that are fit for network TopologySources.
    • To assign instance-level ERI and ERT properties (ILPs) for VMware and/or Kubernetes, please update the following DataSources to their most recent versions:
      1. For VMware:
        1. VMware_vCenter_ClusterPerformance
        2. VMware_vCenter_DatastoreStatus
        3. VMware_vCenter_HostStatus
        4. VMware_vCenter_Vmdk
        5. VMware_vCenter_VMStatus
      2. For Kubernetes clusters:
        1. Kubernetes_Container

      Note: For ERIs defined in DataSources, we are leveraging active discovery (AD) and using batchscript to gather ERIs in order to reduce the number of API calls.

  2. From the repository, import all TopologySources. As discussed in the TopologySources section of this article, TopologySources are a new type of LogicModule created for topology mapping.


    Note: As with DataSources, TopologySources are created to run out of the box and rarely require any custom configurations. To this end, all AppliesTo() functions  are already appropriately set and require no modifications.

  3. To ensure that TopologySource scripts execute successfully, update all Collectors to version 28.000 or higher. Collectors on versions < 28.000 are not able to accept the payload that TopologySources generate.

Once you’ve completed these steps, your newly-imported TopologySources will automatically start connecting ERI keys and forming relationships among all resources currently monitored by LogicMonitor in order to dynamically generate topology maps. Topology maps can be created, saved, and added to dashboards from the Mapping page, as discussed in Mapping Page.

Note: By default, only users assigned the default administrator and manager roles will initially be able to render topology maps. For more information on the permissions available for topology mapping, see Role-based Access Control for Topology Mapping Features section of this support article.

Components of Topology Mapping

There are several primary components that contribute to the generation of a topology map:

Vertices

Vertices are logical objects inside LogicMonitor that represent one or more resources within topology. A vertex is identified by the external resource ID (ERI) and external resource type (ERT) that is assigned to the underlying resource being represented by the vertex.

External Resource IDs (ERIs) and External Resource Types (ERTs)

As discussed in the following sections, ERIs and ERTs are properties set on devices, instances, and services (collectively referred to as resources on the LogicMonitor platform) that allow resources to be recognized for inclusion in topology maps.

ERIs

An ERI contains a set of keys that, in combination, serve to uniquely identify a resource. One ERI can consist of many keys (all keys as stored as CSVs within the ERI itself) and a key represents one of three types of identifiers:

  • Media access control (MAC) address
  • Internet protocol (IP) addresses
  • Fully qualified domain names (FQDNs)

One of the key characteristics of an ERI is that it is truly external to LogicMonitor and does not contain any information that is LogicMonitor-specific (such as device ID). This is critical because connecting disparate resources across a network requires different technology domains to communicate with each other. For example, in order to create a layer 2 map of your network, you will need the MACs of all the devices in your infrastructure. Some devices may expose the MAC via SNMP, some via WMI, and perhaps some only via an API. In this scenario, the MAC addresses would be the keys in the ERI property. The ability for one ERI to contain many keys (stored as CSVs within the ERI itself) supports the scenario, for example, where the MAC address of every interface on a switch is needed to create a topology map of VLANS.

ERI Merging

Another benefit of using an ID external to LogicMonitor is the ability to identify and merge duplicate resources. Because LogicMonitor has the flexibility to monitor a resource multiple times, often with different technology domains (e.g. SNMP/WMI or vCenter for a VM), there needs to be a way to represent that resource cohesively. Without the concept of a vertex (which is identified by an ERI), the same resource could be represented twice on a topology map. To avoid this, LogicMonitor looks for duplicate ERI keys when rendering topology maps; if the same ERI key is assigned to two or more resources, they are merged into a single vertex. In the event that a vertex represents merged resources, its display name will be that of the resource with the smallest ERI and its ERT will be the one that comes first alphabetically.

In rare instances, ERI merging can cause issues when ERI keys match, but not for the above-stated expected reason. For example, there could be a network loopback adapter on a Windows host that shares the same MAC address with all Windows hosts. Since the same ERI PropertySource is being applied to all Windows hosts, they would then all have an identical MAC address keys. This would cause a massive merging and, upon reaching an established maximum threshold of ERI keys, LogicMonitor’s backend would reject the payload in order to prevent performance issues. As a result, the topology map UI would not display any relationships. If this issue occurs in your environment, you can enter the unwanted duplicate ERI key into the IgnoredMacs field found in the PropertySource.

Two commands are available from the Collector Debug Facility that allow you to debug issues related to the unintended existence of duplicate external resource IDs (ERIs):

  • !erimergelist [threshold=xxx] - This command lists all ERI values that appear across resources/instances (or those that meet a designated threshold if specified)
  • !erimergedetail - This command lists all the resources/instances bound to an ERI value

ERTs

While an ERI specifies the unique identifier of a resource/vertex, the external resource type (ERT) specifies its type. The value assigned to an ERT determines the technology-specific icon used to represent the resource/vertex on rendered topology maps.

Setting ERIs and ERTs

Both the ERI and ERT for a resource are defined by either a PropertySource (for device-level ERI properties) or a DataSource (for instance-level ERI properties). With the exception of resources that are members of service groups (i.e. LM Service Insight), one resource will only have one ERI/ERT assigned. Resources that are members of service groups will additionally have ERIs that were auto-generated by LogicMonitor solely for the purpose of connecting them to their service. In the scenario where a resource has a service group ERI and an ERI set by either a PropertySource or DataSource, the ERI that is set by the PropertySource or DataSource takes priority.

Edges

In Topology, an edge represents the relationships between two vertices on a topology map. Any given vertex can have multiple edges. In most scenarios, a vertex will have at least two edges: one that is outgoing and one that is incoming.

An outgoing edge indicates that the respective vertex has relationships that are emanating outward. For example, imagine that you want to view a Cisco switch’s CDP neighbors. To do this, you would select the outgoing edge labeled “NETWORK” on the topology map and all CDP neighbors would appear. Conversely, an incoming edge means that the respective vertex has been discovered by a TopologySource applied to a different resource. In the scenario provided, all of the vertices that were initially discovered by the vertex with the outgoing edge "NETWORK" would also have incoming edges named "NETWORK."

The name of the edge that connects two resources is the edge type. If we continue with the “NETWORK” example, “NETWORK” is the edge type. For every edge, there can only be one edge type. There cannot be multiple names for the same edge between two resources.

The edge and the edge type are both defined in the TopologySource. In the JSON payload produced by the TopologySource script, the edge type is represented as "Type."

TopologySources

TopologySources use ERIs to define the connections between vertices. TopologySources are a new type of LogicModule introduced specifically for topology mapping and take a form similar to scripted DataSources. The successful output of a TopologySource is a JSON object consisting of vertices and edges.

For most networking topology, TopologySources are leveraging layer 2 discovery protocols (i.e. LLDP and CDP) to gather information about VLANs. This information is contained in certain OIDs, WMI classes, and API calls. For VMware, TopologySources are accessing the vCenter API to gather relationship information and leverage the VMMoRefIds.

Note: In rare instances, TopologySources can execute “successfully,” but no resulting relationships display in the mapping UI. This is because TopologySources don’t technically need ERIs defined on AppliedTo resources in order to successfully execute. If you experience this, it is likely that you have not imported all the necessary files (i.e. PropertySources and DataSources) responsible for assigning ERIs to the resources currently monitored by LogicMonitor. See the Getting Started section of this support article for a detailed listing of required files.

An Example Scenario to Tie It All Together

The following example scenario was created to help you visualize how TopologySources use ERIs to define connections between resources and render topology maps.

In the diagram above, we are illustrating how a TopologySource orchestrates the relationship between an airplane and the hanger that it parks in. Let’s walk through the diagram counter-clockwise:

  1. Starting with the TopologySource in the bottom right of the diagram, we see that its “recipe” is to connect resources whose ERIs contain the key “123333” with resources whose ERIs contain the key “VanGogh.” Once these resources are connected, an edge (relationship) is formed with the edge type (i.e. name) of “Parking-In.”
  2. Immediately above the TopologySource, three resources have been identified as having ERIs containing the keys referenced by the TopologySource: Main Hangar, Boeing, and Airbus. Two of these resources have an external resource type (ERT) of “Airplane” and one has an ERT of “Hangar.” Three different DataSources assigned these ERI/ERTs respectively.
  3. As we move to the left, we see how the topology map is modeled on the backend. Notice in the upper left that there is only one vertex representing the two airplane resources. As discussed in the ERI Merging section of this support article, LogicMonitor looks for duplicate ERI keys when rendering topology maps; if the same ERI key is assigned to two or more resources (as is the case here, both share the ERI key “Riverland”) they are merged into a single vertex. In the event that a vertex represents merged resources, its display name will be that of the resource with the smallest ERI and its ERT will be the one that comes first alphabetically.
  4. Continuing on, the Main Hangar resource is modeled, without change, as a vertex.
  5. And, finally, the end result of the diagram is the connection between the Airbus and its Main Hangar, defined by an edge named “Parking-In.”

Extrapolating Example to VMware

Next, let’s extrapolate from the simplistic, non-real-world example highlighted previously to one that is likely more pertinent to your LogicMonitor operations: mapping topology among VMware components (e.g. vCSA → cluster → host → VM → etc.). In the case of VMware, DataSources will assign both the managed object reference ID (MoRef) and MAC address as resource ERI keys.

For example, the result of the ERI script would look something similar to predef.externalresourceid = 10.41.16.7--host-1981, 00:21:28:6b:09:dc, where the host ID is appended to the vCenter's IPv4 address and the MAC address is also assigned to the resource as an ERI key. Assigning the MoRef as an ERI key allows you to leverage the vCenter API to define multiple relationships; the MAC address is used to tie the host back into the network and show the switch to which it is connected.

The TopologySource would look to connect ERI keys assigned to virtual machines, hypervisors, cluster, vCSAs, and so on. If LogicMonitor detected any duplicate ERI keys among these components (e.g. you’re monitoring a VM via vCenter, and also via SNMP as a separate resource), they would be merged into a single vertex when rendered by a topology map.

Role-based Access Control for Topology Mapping Features

Topology mapping, like many other features in the LogicMonitor platform, supports role-based access control. By default, only users assigned the default administrator and manager roles will be able to render topology maps and access the full range of topology mapping features (users assigned the readonly role will see the Mapping page and any saved maps). As discussed in Roles, roles can be created/updated to allow for full or limited access to topology mapping features.

Creating Custom Topology Mapping Relationships

For the vast majority of users, there will be no need to create custom TopologySources or manually assign ERIs and ERTs, assuming all relevant files have been imported. Topology is automatically discovered and topology maps are dynamically generated. However, similar to DataSources, the LogicMonitor platform does expose the ability to create new (or edit existing) TopologySources. Similarly, DataSources and/or PropertySources can be created or modified to assign custom ERIs and ERTs. Because this is an involved process, requiring careful consideration in ensuring unique ERIs, as well as script building, we recommend you reach out to our support team for guidance.

FAQs

Q: “Test Script” isn’t working properly on the TopologySource.
A:
 Make sure Collectors are running version 28.000 or better.

Q: My TopologySource is reporting edges, but the UI is not reflecting the same.
A:
Make sure an ERI is applied and is hitting the correct resources. TopologySources report relationships independent of ERI scripts which may confuse users into thinking that topology should be available on the UI as well.

Q: My TopologySources are not executing successfully even though I have the proper credentials set.
A:
For SNMP-based TopologySources, it has been observed that some OIDS cannot be queried for on a device. Example: Querying a Juniper Switch for CDP (Cisco Discovery Protocol).

Q: My network topology does not relate to how my network actually is laid out.
A:
Make sure that all resources that are in the network path are being monitored by LogicMonitor, and that they have the proper credentials (e.g. snmp, wmi, esx.user, and esx.pass) and protocols enabled.

Q: I am seeing a vertex with a different display name than what I was expecting.
A:
This is likely due to a merging of resources as a result of one or more resources having the same ERI key. Learn more about ERI merging in the ERI Merging section of this support article.

Q: I don’t want my users to see the Maps tab from the detail view of a resource. How can I restrict this?
A:
When creating/editing roles, there is an Allowed to view Maps Tab option under the Resource heading that, when unchecked, will prevent the Maps tab from displaying.

Q: What are the default view settings for all topology features?
A:
By default, only default administrator and manager roles will be able to render topology maps. As discussed in the Creating Custom Topology Mapping Relationships section of this support article, admins can add/restrict permissions to topology features for other roles/users as they see fit.

Q: Is topology automatically discovered?
A:
Yes. Once all necessary LogicModules have been downloaded, as outlined in the Getting Started section of this support article, topology discovery is handled by TopologySources and relationships are auto generated.

Q: Why doesn’t LogicMonitor auto-create a map of my entire environment?
A:
Our mapping philosophy is based on an additive approach, which only shows what is necessary to perform the task at hand (i.e. troubleshooting, identifying a certain edge, etc.). This is opposed to a subtractive model, where every infrastructure element is initially displayed, requiring the user to filter down to find what is relevant. Our end goal with topology is to enable efficient troubleshooting of incidents as they arise, and provide meaningful context that can be used to facilitate root cause analysis.

Q: Do I need to add a map for each resource?
A:
No. A single map can be modeled to contain any logical grouping of resources you see fit. This could be a customer site, a particular business unit, the supporting infrastructure a service runs on, and so on.

Q: What environments are we able to monitor with topology? Does this extend to cloud?
A:
Due to the extensibility of LogicMonitor, TopologySources can be written to map any type of resource, provided it exposes metrics in a way that LogicMonitor can capture.

Q: Is the initial release of topology only level 2 mapping for network? Do I need to enable CDP or LLDP?
A:
Yes. Our primary focus was to map interswitch connectivity, and how the rest of the IT fabric relates to the switch. The most feasible route to accommodate this was to use relationships on level 2. And for this to work most optimally, CDP and LLDP will need to be enabled.

Q:  Do I need to enable CDP/LLDP on my firewalls?
A:
While the results of using CDP and LLDP are far more reliable and accurate, the only resources that must be CDP/LLDP-enabled are switches and routers. The TopologySource determining the connection should be intelligent enough to connect to the firewall using bridging table information (MACs) and not require CDP/LLDP.

Q: Will the load generated by the TopologySource place excess strain on my Collectors?
A:
Since we are gathering relationship data with protocols such as SNMP, WMI, API, etc., there is a necessary increase in Collector load. However, since the data we are gathering is not metric data, the polling interval can be greatly reduced (e.g. >1 hour in some scenarios).

Q: Are there any aspects of LogicMonitor’s topology capabilities that will entail add-on costs?
A:
Some aspects of topology, such as the number of saved maps, the Topology dashboard widget, and other advanced overlays planned for future releases are currently considered “preview” functionality. We anticipate that they will be subject to charge/limitation in several months’ time. We will communicate any changes in topology pricing to accommodate budget planning and renewal efforts.

Q: Can I manually define topologies in the UI?
A:
Manually-defined topology maps is a feature that we are actively working on; it will be announced when ready. In the meantime, users can take advantage of LM Service Insight to connect otherwise disparate resources on a topology map. In addition to this, a custom TopologySource can allow for the ability to manually define relationships.