LogicModules

Interpretation Methods for Normal Datapoints

The data collected from your devices can sometimes be used directly (e.g. the responseTime of the web page collector) but other times requires further interpretation.  

Interpret output with

Designates the datapoint post-processor method. This informs LogicMonitor how to extract the desired values from the raw output collected.

If you select "No Interpretation: use directly," then the raw standard output will be used as the datapoint value.  Typically, SNMP datasources use this method.

If you select No Interpretation: use directly as the interpretation method, then the raw standard output will be used as the datapoint value.  Typically SNMP datasources use this method - the numeric output of the OID is the data that should be stored and monitored.

Methods Available

The following is a list of supported methods (although not all methods apply to all collection methods):

Multi-Line Key-Value Pairs

Treats a multi-line string raw measurement as a set of key-value pairs separated by an equal sign or colon.  E.g.

Buffers=11023
BuffersAvailable=333
heapSize=245MB

You can extract the total number of Buffers from the above output by specifying:

Note: Key names should be unique for each Key-Value Pair.  If a raw measurement output contains two identical key names paired with different values, where the separating character (equal sign or colon) differs, the Key-Value Pairs Post Processor method will not be able to extract the values. 

Regular Expressions

You can use regular expressions to extract data from more complex strings. The contents of the first capture group - i.e. text in parenthesis - in the regex will be assigned to the datapoint. For example, the Apache datasource uses a regular expression to extract counters from the server-status page. The raw output of the webpage collector looks like:

Total Accesses: 8798099
Total kBytes: 328766882
CPULoad: 1.66756
Uptime: 80462

To extract the Total Accesses, you could define a datapoint as follows:

TextMatch

Look for presence of a string (TextMatch) tests whether a string is present within the raw output. This method supports regular expressions for text matching.  If the string exists, it returns 1, otherwise 0.

For example, to check if Tomcat is running on a host, you may have a script datasource that executes 'ps -ef | grep java' periodically. The output from the pipeline should contain "org.apache.catalina.startup.Bootstrap start" if Tomcat is running.

The datapoint below checks if the raw measurement output contains  the string "org.apache.catalina.startup.Bootstrap start".  If yes, the datapoint will be 1 indicating Tomcat is running, otherwise the datapoint value will be 0.

TextMatch

CSV and TSV

If the raw measurement is an array of comma-separated values (CSV) or of tab-separated values (TSV), you can use the CSV and TSV methods to extract values, respectively.

There are three forms of parameters that the post processor will accept for CSV and TSV methods:

  • a simple integer N - extract the Nth element of the first line of input (elements start at zero)
  • index=Nx,Ny - extract the Nxth element of the Nyth line of the input
  • line=regex index=Nx - extract the Nxth element of the first line which matches the given regex

For example, a script datasource executes 'iostat | grep 'sda' | head -1' to get the statistics of hard disk sda. The output is:

sda              33.75         3.92       719.33    9145719 1679687042

which is a TSV array.  The fourth column (719.33) is blocks written per second. To extract this value into a datapoint using the TSV method, select "a TSV string" as the method to interpret the output, and 3 as the index, like so:

HexExpr

The 'Hex String with Hex Regex Extract Value' (Hexexpr) method only applies to the TCP and UDP collector methods.  It is applied to the TCP and UDP payloads returned by these collector methods.

The payload is treated as a byte array. You specify an offset and a length that could be 1, 2, 4, or 8 (corresponding to data type byte, short, int, or long, respectively) in the format of offset:length in the HexExpr Regex field. Hexexpr will return the value of the byte (short, int, or long) starting at offset of the array.

For for interpreting 32-bit values, you can also specify the underlying byte ordering with the following method selections:

  •  A Big-Endian 32-bit Integer Hex String with Hex Regex Extract Value
  • A Little-Endian 32-bit Integer Hex String with Hex Regex Extract Value

The Hexexpr method can be very useful if you'd like a datapoint to return the value of a field in binary packets such as DNS.

XML

The XML Document interpretation uses XPath syntax to navigate through elements and attributes in an XML document.

For example, if the web page collector was used to collect the following content:

<Order xmlns="https://www.example.com/myschema.xml">
<Customer>
 <Name>Bill Buckram</Name>
 <Cardnum>234 234 234 234</Cardnum>
</Customer>
<Manifest>
 <Item>
  <ID>209</ID>
  <Title>Duke: A Biography of the Java Evangelist</Title>
  <Quantity>1</Quantity>
  <UnitPrice>12.75</UnitPrice>
 </Item>
 <Item>
  <ID>204</ID>
  <Title>
  Making the Transition from C++ to the Java(tm) Language
  </Title>
  <Quantity>1</Quantity>
  <UnitPrice>10.75</UnitPrice>
 </Item>
</Manifest>
</Order>

You could extract the order ID for the book "Making the Transition from C++ to the Java Language" from the XML page above by specifying in the datapoint configuration that the HTTP response body should be interpreted with an XML document and setting the Xpath to: Order/Manifest/Item[normalize-space(Title)="Making the Transition from C++ to the Java(tm) Language"]/ID

Any Xpath expression that returns a number is supported.  Other examples that could be used to extract data from the sample XML above:

  • count(/Order/Manifest/Item)
  • sum(/Order/Manifest/Item/UnitPrice)

JSON

Some collectors return a JSON literal string as the value of one of the raw datapoints. For example, you can create a webpage datasource sending a HTTP request "/api/overview" to RabbitMQ server to collect its performance information. RabbitMQ returns a JSON literal string like this:

{
  "management_version":"2.5.1",
  "statistics_level":"fine",
  "message_stats":[],
  "queue_totals":{
    "messages":0,
    "messages_ready":0,
    "messages_unacknowledged":0
  },
  "node":"[email protected]",
  "statistics_db_node":"[email protected]",
  "listeners":[
    {
      "node":"[email protected]",
      "protocol":"amqp",
      "host":"labpnginx01",
      "ip_address":"::",
      "port":5672
    }
  ]
}

You could create a datapoint that uses the JSON/BSON post-processor to extract the number of messages in the queue like this:

Members can be retrieved using dot or subscript operators.  LogicMonitor uses the syntax used in JavaScript to identify objects.  For example, "listeners[0].port" will return 5672 for the raw data displayed above.

LogicMonitor supports JSON Path to select objects or elements.  A comparison of Xpath and JSON Path syntax elements:

Xpath

JSON Path

Description

/ $ the root object/element
. @ the current object/element
/ . or [] child operator
.. n/a parent operator
// .. recursive descent. JSON Patch borrows this syntax from E4X
* * wildcard. All objects/elements regardless of their names.
@ n/a attribute access. JSON structures don't have attributes.
[] [] subscript operator. Xpath uses it to iterate over element collections and for predicates. In Javascript and JSON it is the native array operator.
| [,] Union operator in Xpath results in a combination of node sets. JSON Path allows alternate names or array indices as a set.
n/a [start:end:step] array slice operator borrowed from ES4.
[] ?() applies a filter (script) expression.
n/a () script expression, using the underlying script engine.
() n/a grouping in Xpath

 

JSON Aggregate Functions:

The JSON interpretation method supports four aggregation operators that work on JSON paths:

[sum|average|min|max](JsonPath)

For example, using the following JSON:

{
     "store":{
        "book":[
          {
               "title":"Harry Potter and the Sorcerer's Stone",
               "price":"10.99"
          },
          {
               "title":"Harry Potter and the Chamber of Secrets",
               "price":"10.99"
          },
          {
               "title":"Harry Potter and the Deathly Hallows",
               "price":"9.99"
          },
          {
               "title":"Lord of the Rings: The Return of the King",
               "price":"17.99"
          },
          {
               "title":"Lord of the Rings: The Two Towers",
               "price":"17.99"
          }
        ]
     }
}

To retrieve the total price of all the books, you could use the aggregate function:

sum($.store.book[*].price)

To break this down: sum is the aggregate function. $ is the start of JsonPath. * is a wildcard, to return all objects regardless of their names.  Then the price element for each object is returned.

The aggregate functions can only be used in the outermost layer.

Regex filter  =~

The Regex Filter =~ can be used to select all elements that match the specified regex expression. For example, to calculate the sum of all the Harry Potter series, you could use the expression:

This expression can be broken down as the sum of the price of all books in the store that pass the filter expression: those whose title match the regex "Harry Potter'. You can use regex match only in filter expressions.

“@” means the current object which represents each book object in the book array in the example.